From christophe.ortiz at ciemat.es  Tue Oct  1 02:18:55 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 1 Oct 2013 09:18:55 +0200
Subject: [petsc-users] Back to struct in Fortran to represent field with
 dof > 1
In-Reply-To: <214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
References: <CANBrw+52SARdE0Nr0RUh094nd=CQ=UkApr3sgHNvaqihjdqT7w@mail.gmail.com>
	<B395E1EA-628A-417D-94EC-5A62C653C916@mcs.anl.gov>
	<CANBrw+7O+8MZQ1F1ZbJeF8CZAQeSFUCsusYHbPvo--5JzAV+pQ@mail.gmail.com>
	<CANBrw+7=1N=Cd14x30MppriShh-_NypuV2SLbj4aUe_ApB6bVA@mail.gmail.com>
	<3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov>
	<CANBrw+4ugcmFORuZhDy2sCG86P_GxyePa3u8CNMqNBvv6cnqxw@mail.gmail.com>
	<B77F0A05-850C-42B2-9451-614D732F940C@mcs.anl.gov>
	<CANBrw+6yXPgR8FU1v3qsqQYJJi6s+7ZSUiPaZuverR4sefy3jA@mail.gmail.com>
	<DED33F6A-35AB-4422-9612-6E3A32F8D6AC@mcs.anl.gov>
	<CANBrw+7tocNMkaFdyQ5xYVhepy-_7fmTPihK=8_ExGOVfJjYvA@mail.gmail.com>
	<A4273914-DCAB-4B2A-B0B7-B457898DDFE1@mcs.anl.gov>
	<CANBrw+49+1jcKmUtFHmP-50x_mecDmpnx4bfGWKDk9f0j04DxQ@mail.gmail.com>
	<33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov>
	<CANBrw+4uxe7oR7Du0MKEa=nNKQV8PqGq0HhO=K183m9ScH0Lrg@mail.gmail.com>
	<214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
Message-ID: <CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>

Hi Barry,

Thanks for the piece of code. I will try to implement it. My problem is 1D,
so should be ok. If I have some pb I'll let you know.

If this work, do you plan to include it in next version of PETSc ?
Christophe

On Mon, Sep 30, 2013 at 2:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Sep 30, 2013, at 3:01 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> > Hi Barry,
> >
> > It works ! Great job ! Here is the output:
> >
> >
> > Components have the right values.
> >
> > Do you think it can be adapted to work with DMDAVecGetArrayF90(da,...) /
> DMDAVecRestoreArrayF90(da,...) in order to work with distributed arrays ?
>
>   What dimension do you need?  For one dimension, which I think you
> mentioned before it is almost identical to what I sent you
>
> PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec
> *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> {
>   PetscInt    xs,ys,zs,xm,ym,zm,gxs,gys,gzs,gxm,gym,gzm,N,dim,dof;
>   PetscScalar *aa;
>
>   *ierr = DMDAGetCorners(*da,&xs,&ys,&zs,&xm,&ym,&zm);if (*ierr) return;
>   *ierr = DMDAGetGhostCorners(*da,&gxs,&gys,&gzs,&gxm,&gym,&gzm);if
> (*ierr) return;
>   *ierr = DMDAGetInfo(*da,&dim,0,0,0,0,0,0,&dof,0,0,0,0,0);if (*ierr)
> return;
>
>   /* Handle case where user passes in global vector as opposed to local */
>   *ierr = VecGetLocalSize(*v,&N);if (*ierr) return;
>   if (N == xm*ym*zm*dof) {
>     gxm = xm;
>     gym = ym;
>     gzm = zm;
>     gxs = xs;
>     gys = ys;
>     gzs = zs;
>   } else if (N != gxm*gym*gzm*dof) {
>     *ierr = PETSC_ERR_ARG_INCOMP;
>   }
>   *ierr = VecGetArray(*v,&aa);if (*ierr) return;
>   *ierr = F90Array1dCreate(aa,PETSC_SCALAR,gxs,gxm,a
> PETSC_F90_2PTR_PARAM(ptrd));if (*ierr) return;
> }
>
>
> except it (1) handles both the local vector or global vector version (the
> if above) and (2) handles that start index at gxs instead of 0.  All you
> should have to do is rename the beast above, provide an FORTRAN 90
> interface for it, have it call the specialized version of F90Array1dCreate
> I already provided.   Two and three dimensions are a bit more involved
> since you need to copy over and modify more functions but I don't think
> there are any tricky things.
>
>
>    Barry
>
>
> >
> > Best regards,
> > Christophe
> >
> >
> > 2013/9/29 Barry Smith <bsmith at mcs.anl.gov>
> >
> >
> >    Sorry, that is what I get for having a ex1.F and ex1F90.F in the same
> directory.
> >
> >    Barry
> >
> > On Sep 29, 2013, at 6:16 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> >
> > > Hi Barry,
> > >
> > > I just looked at exf1.F. It seems you sent me a wrong file. It does
> not correspond to ex1f90. There is no vecgetarraymystruct in the file you
> sent me. Could you please send me the correct version of ex1f90 so I can
> test it ? Many thanks in advance.
> > >
> > > Christophe
> > >
> > > CIEMAT
> > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > > Unidad de Materiales
> > > Edificio 2 - Planta 0 - Despacho 28m
> > > Avenida Complutense 40,
> > > 28040 Madrid, Spain
> > > Tel: +34 91496 2582
> > > Fax: +34 91346 6442
> > >
> > > --
> > > Q
> > > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > > Please consider the environment before printing this email.
> > >
> > >
> > > 2013/9/27 Barry Smith <bsmith at mcs.anl.gov>
> > >
> > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > >
> > > > The files were not attached. Could you please send them again ?
> > > >
> > > > Thanks.
> > > > Christophe
> > > >
> > > > El 27/09/2013 21:28, "Barry Smith" <bsmith at mcs.anl.gov> escribi?:
> > > >
> > > >    Please find attached a new version of the code. ex1.c and
> ex1f90.F please run it as is and send me all the output.
> > > >
> > > >    It works correctly with gfortran and the intel 12.x ifort
> compiler, I believe it should now work for all fortran compilers. If it
> does not work as expected please send me the name and version of your
> fortran compiler.
> > > >
> > > >    Barry
> > > >
> > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > > >
> > > > > There was a misunderstanding. There are n entries in total in the
> vector. I thought it was a vector of n points, with 3 components associated
> to each point, ie 3*n entries. This is clear now.
> > > > >
> > > > > I've tested your example and I checked the values assigned to the
> components with your piece of code:
> > > > >
> > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > >       do i=1,10
> > > > >       xarray(i)%a = i
> > > > >       xarray(i)%b = 100*i
> > > > >       xarray(i)%c = 10000*i
> > > > >       enddo
> > > > >
> > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > >
> > > > >
> > > > > I did it with:
> > > > >
> > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > >        do i = 1 , 10
> > > > >         write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c
> > > > >        end do
> > > > >
> > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > >
> > > > > I obtained the following:
> > > > >
> > > > >    1.00000000000000        2.00000000000000        3.00000000000000
> > > > >    2.00000000000000        3.00000000000000        4.00000000000000
> > > > >    3.00000000000000        4.00000000000000        5.00000000000000
> > > > >    4.00000000000000        5.00000000000000        6.00000000000000
> > > > >    5.00000000000000        6.00000000000000        7.00000000000000
> > > > >    6.00000000000000        7.00000000000000        8.00000000000000
> > > > >    7.00000000000000        8.00000000000000        9.00000000000000
> > > > >    8.00000000000000        9.00000000000000        10.0000000000000
> > > > >    9.00000000000000        10.0000000000000        1000.00000000000
> > > > >    10.0000000000000        1000.00000000000        100000.000000000
> > > > >
> > > > > First column if ok (1st component), but the other two columns show
> wrong numbers. For component 2 and 3, only the very last value is ok. It
> seems values from component b are copied from a, and those from component c
> from b, with an offset.
> > > > >
> > > > > What do you think ?
> > > > >
> > > > > Christophe
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >
> > > > >   Indices start in fortran arrays from 1 by default so don't even
> try zero
> > > > >
> > > > >    I've attached a new version that has three components,
> > > > >  30 TOTAL vector entries and 10 for each component. Please run it
> and send the results back.
> > > > >
> > > > >     Barry
> > > > >
> > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > > > >
> > > > > > Hi Barry,
> > > > > >
> > > > > > I made some tests with your code that allows to use struct in
> Fortran90. It compiles and does not complain with the struct with 2
> components but...it exhibits some strange behaviour. Here are the
> behaviours I observed. To make it simple, I used n=10.
> > > > > >
> > > > > > - After using call VecSet(x,one,ierr), I checked the values of
> each component of the vector using
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >       do i = 0 , n
> > > > > >        write(*,*) i , xarray(i)%a, xarray(i)%b
> > > > > >       end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > > I checked the values from i= 0 to n since I was not sure what
> was the convention (0- or 1-based). I obtained:
> > > > > >
> > > > > >            0  0.000000000000000E+000   1.00000000000000
> > > > > >            1   1.00000000000000        1.00000000000000
> > > > > >            2   1.00000000000000        1.00000000000000
> > > > > >            3   1.00000000000000        1.00000000000000
> > > > > >            4   1.00000000000000        1.00000000000000
> > > > > >            5   1.00000000000000        1.00000000000000
> > > > > >            6   1.00000000000000        1.00000000000000
> > > > > >            7   1.00000000000000        1.00000000000000
> > > > > >            8   1.00000000000000        1.00000000000000
> > > > > >            9   1.00000000000000        1.00000000000000
> > > > > >           10   1.00000000000000       1.996650376645798E-314
> > > > > >
> > > > > >
> > > > > > As you can see, for component a, there is a 0 for i=0, then the
> values are correct. For component b, correct values go from i=1 to 9.There
> is garbage in second component for i=10 . I checked that if you define a
> third component c, its values go from i=-1 to n-2. It's like if values of
> second component started at the end of first component, and same thing for
> third component.
> > > > > >
> > > > > > Then, instead of using VecSet(), I tried to assign values
> directly to each component of the vector with
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >       do i = 0 , n-1
> > > > > >        xarray(i)%a = 2.0
> > > > > >        xarray(i)%b = 3.0
> > > > > >       end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > > Here I checked that indices actually start at 0 and not at 1.
> With a loop from i=1 to n I got a Memory corruption message.
> > > > > >
> > > > > > and I checked that the values are correctly assigned with
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >        do i = 0 , n-1
> > > > > >         write(*,*) i, xarray(i)%a, xarray(i)%b
> > > > > >        end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Then, I obtained:
> > > > > >
> > > > > >            0   2.00000000000000        2.00000000000000
> > > > > >            1   2.00000000000000        2.00000000000000
> > > > > >            2   2.00000000000000        2.00000000000000
> > > > > >            3   2.00000000000000        2.00000000000000
> > > > > >            4   2.00000000000000        2.00000000000000
> > > > > >            5   2.00000000000000        2.00000000000000
> > > > > >            6   2.00000000000000        2.00000000000000
> > > > > >            7   2.00000000000000        2.00000000000000
> > > > > >            8   2.00000000000000        2.00000000000000
> > > > > >            9   2.00000000000000        3.00000000000000
> > > > > >
> > > > > >
> > > > > > Here the problem seems more severe. Values are not correctly
> assigned to the second component. There should be 3.0 in the second column.
> It seems values of the first component are copied to the second one. Only
> for i=10 the value of xarray(i)%b is correct (3.0).
> > > > > >
> > > > > > Any idea where it could come from ?
> > > > > > I guess it needs some fixes here and there but I think your idea
> is good and that it could work.
> > > > > >
> > > > > > Christophe
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > > >
> > > > > >    I just put it with the Fortran source code and compile it
> with the rest of the application code; here is the makefile I used
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > > > > >
> > > > > > > By the way, what should I do with the small .c code ? Where
> should I put it ?
> > > > > > >
> > > > > > > Christophe
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith <
> bsmith at mcs.anl.gov> wrote:
> > > > > > >
> > > > > > >    Christophe,
> > > > > > >
> > > > > > >      Despite my LinkIn Endorsement for expertise in Fortran
> :-) I cannot pretend to be an expert in FortranXX but I have cooked up an
> example demonstrating accessing the Vec entries as if they are in an array
> of derived types. I've attached the example code; there needs to be a small
> C stub that defines the functions for your specific derived type name.
> > > > > > > Note that it will only work I think if your N is a compile
> time constant.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >     It worked with
> > > > > > > ~/Downloads$ gfortran --version
> > > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease)
> > > > > > >
> > > > > > >
> > > > > > >     I do not understand exactly why it works since it uses
> F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd));
> which has a single PETSC_SCALAR as a building block but ? I hope it works
> for you. If it doesn't, let us know the compiler you are using and we may
> be able to get it working for that compiler.
> > > > > > >
> > > > > > >    Barry
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Me again !
> > > > > > > >
> > > > > > > > I have read in previous posts that it is hard in Fortran to
> declare something similar to a typedef struct in C to manage a
> multicomponent problem.
> > > > > > > >
> > > > > > > > Is it still the case ? Has the problem been solved ?
> > > > > > > >
> > > > > > > > I am asking because my plan is to implement a multicomponent
> problem (in 1D), with many components that will be organized in arrays of
> two dimensions. In C I could define
> > > > > > > >
> > > > > > > > typedef struct{
> > > > > > > > PetscScalar U;
> > > > > > > > PetscScalar V;
> > > > > > > > PetscScalar A[N][N];
> > > > > > > > } Field;
> > > > > > > >
> > > > > > > > and then I could calculate the residual function with
> > > > > > > >
> > > > > > > > F[i].U = ...
> > > > > > > > F[i].V = ...
> > > > > > > > F[i].A[k][n] = ...
> > > > > > > >
> > > > > > > > which is quite convenient.
> > > > > > > >
> > > > > > > > If in Fortran it is not possible to use a struct as in C, I
> am afraid I'll have to deal with
> > > > > > > >
> > > > > > > > F(jdof,i) = ...
> > > > > > > >
> > > > > > > > where I will have only jdof to address U, V and A[ ][ ],
> which can be difficult and not very convenient I guess. Before I move all
> my code to C, does anyone have an alternative idea to manage this
> multi(many)component problem in Fortran ?
> > > > > > > >
> > > > > > > > Many thanks in advance for your help and suggestion !
> > > > > > > >
> > > > > > > > Christophe
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/f7748338/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Oct  1 06:50:09 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Oct 2013 06:50:09 -0500
Subject: [petsc-users] Back to struct in Fortran to represent field with
	dof > 1
In-Reply-To: <CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>
References: <CANBrw+52SARdE0Nr0RUh094nd=CQ=UkApr3sgHNvaqihjdqT7w@mail.gmail.com>
	<B395E1EA-628A-417D-94EC-5A62C653C916@mcs.anl.gov>
	<CANBrw+7O+8MZQ1F1ZbJeF8CZAQeSFUCsusYHbPvo--5JzAV+pQ@mail.gmail.com>
	<CANBrw+7=1N=Cd14x30MppriShh-_NypuV2SLbj4aUe_ApB6bVA@mail.gmail.com>
	<3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov>
	<CANBrw+4ugcmFORuZhDy2sCG86P_GxyePa3u8CNMqNBvv6cnqxw@mail.gmail.com>
	<B77F0A05-850C-42B2-9451-614D732F940C@mcs.anl.gov>
	<CANBrw+6yXPgR8FU1v3qsqQYJJi6s+7ZSUiPaZuverR4sefy3jA@mail.gmail.com>
	<DED33F6A-35AB-4422-9612-6E3A32F8D6AC@mcs.anl.gov>
	<CANBrw+7tocNMkaFdyQ5xYVhepy-_7fmTPihK=8_ExGOVfJjYvA@mail.gmail.com>
	<A4273914-DCAB-4B2A-B0B7-B457898DDFE1@mcs.anl.gov>
	<CANBrw+49+1jcKmUtFHmP-50x_mecDmpnx4bfGWKDk9f0j04DxQ@mail.gmail.com>
	<33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov>
	<CANBrw+4uxe7oR7Du0MKEa=nNKQV8PqGq0HhO=K183m9ScH0Lrg@mail.gmail.com>
	<214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
	<CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>
Message-ID: <4B31F44E-CE2D-433D-9738-306446C30F95@mcs.anl.gov>


On Oct 1, 2013, at 2:18 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> Hi Barry,
> 
> Thanks for the piece of code. I will try to implement it. My problem is 1D, so should be ok. If I have some pb I'll let you know.
> 
> If this work, do you plan to include it in next version of PETSc ?

   I put the example with VecGetArray() into petsc-dev and if you send me one with DMDAVecGetArray() I'll add that as well.  Since the derived type has to appear explicitly in the code I can't put something in the library only in examples that people can copy and change.

    Barry

> Christophe
> 
> On Mon, Sep 30, 2013 at 2:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Sep 30, 2013, at 3:01 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> 
> > Hi Barry,
> >
> > It works ! Great job ! Here is the output:
> >
> >
> > Components have the right values.
> >
> > Do you think it can be adapted to work with DMDAVecGetArrayF90(da,...) / DMDAVecRestoreArrayF90(da,...) in order to work with distributed arrays ?
> 
>   What dimension do you need?  For one dimension, which I think you mentioned before it is almost identical to what I sent you
> 
> PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> {
>   PetscInt    xs,ys,zs,xm,ym,zm,gxs,gys,gzs,gxm,gym,gzm,N,dim,dof;
>   PetscScalar *aa;
> 
>   *ierr = DMDAGetCorners(*da,&xs,&ys,&zs,&xm,&ym,&zm);if (*ierr) return;
>   *ierr = DMDAGetGhostCorners(*da,&gxs,&gys,&gzs,&gxm,&gym,&gzm);if (*ierr) return;
>   *ierr = DMDAGetInfo(*da,&dim,0,0,0,0,0,0,&dof,0,0,0,0,0);if (*ierr) return;
> 
>   /* Handle case where user passes in global vector as opposed to local */
>   *ierr = VecGetLocalSize(*v,&N);if (*ierr) return;
>   if (N == xm*ym*zm*dof) {
>     gxm = xm;
>     gym = ym;
>     gzm = zm;
>     gxs = xs;
>     gys = ys;
>     gzs = zs;
>   } else if (N != gxm*gym*gzm*dof) {
>     *ierr = PETSC_ERR_ARG_INCOMP;
>   }
>   *ierr = VecGetArray(*v,&aa);if (*ierr) return;
>   *ierr = F90Array1dCreate(aa,PETSC_SCALAR,gxs,gxm,a PETSC_F90_2PTR_PARAM(ptrd));if (*ierr) return;
> }
> 
> 
> except it (1) handles both the local vector or global vector version (the if above) and (2) handles that start index at gxs instead of 0.  All you should have to do is rename the beast above, provide an FORTRAN 90 interface for it, have it call the specialized version of F90Array1dCreate I already provided.   Two and three dimensions are a bit more involved since you need to copy over and modify more functions but I don't think there are any tricky things.
> 
> 
>    Barry
> 
> 
> >
> > Best regards,
> > Christophe
> >
> >
> > 2013/9/29 Barry Smith <bsmith at mcs.anl.gov>
> >
> >
> >    Sorry, that is what I get for having a ex1.F and ex1F90.F in the same directory.
> >
> >    Barry
> >
> > On Sep 29, 2013, at 6:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> >
> > > Hi Barry,
> > >
> > > I just looked at exf1.F. It seems you sent me a wrong file. It does not correspond to ex1f90. There is no vecgetarraymystruct in the file you sent me. Could you please send me the correct version of ex1f90 so I can test it ? Many thanks in advance.
> > >
> > > Christophe
> > >
> > > CIEMAT
> > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > > Unidad de Materiales
> > > Edificio 2 - Planta 0 - Despacho 28m
> > > Avenida Complutense 40,
> > > 28040 Madrid, Spain
> > > Tel: +34 91496 2582
> > > Fax: +34 91346 6442
> > >
> > > --
> > > Q
> > > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > > Please consider the environment before printing this email.
> > >
> > >
> > > 2013/9/27 Barry Smith <bsmith at mcs.anl.gov>
> > >
> > > On Sep 27, 2013, at 3:06 PM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> > >
> > > > The files were not attached. Could you please send them again ?
> > > >
> > > > Thanks.
> > > > Christophe
> > > >
> > > > El 27/09/2013 21:28, "Barry Smith" <bsmith at mcs.anl.gov> escribi?:
> > > >
> > > >    Please find attached a new version of the code. ex1.c and ex1f90.F please run it as is and send me all the output.
> > > >
> > > >    It works correctly with gfortran and the intel 12.x ifort compiler, I believe it should now work for all fortran compilers. If it does not work as expected please send me the name and version of your fortran compiler.
> > > >
> > > >    Barry
> > > >
> > > > On Sep 27, 2013, at 8:22 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> > > >
> > > > > There was a misunderstanding. There are n entries in total in the vector. I thought it was a vector of n points, with 3 components associated to each point, ie 3*n entries. This is clear now.
> > > > >
> > > > > I've tested your example and I checked the values assigned to the components with your piece of code:
> > > > >
> > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > >       do i=1,10
> > > > >       xarray(i)%a = i
> > > > >       xarray(i)%b = 100*i
> > > > >       xarray(i)%c = 10000*i
> > > > >       enddo
> > > > >
> > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > >
> > > > >
> > > > > I did it with:
> > > > >
> > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > >        do i = 1 , 10
> > > > >         write(*,*) xarray(i)%a,xarray(i)%b,xarray(i)%c
> > > > >        end do
> > > > >
> > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > >
> > > > > I obtained the following:
> > > > >
> > > > >    1.00000000000000        2.00000000000000        3.00000000000000
> > > > >    2.00000000000000        3.00000000000000        4.00000000000000
> > > > >    3.00000000000000        4.00000000000000        5.00000000000000
> > > > >    4.00000000000000        5.00000000000000        6.00000000000000
> > > > >    5.00000000000000        6.00000000000000        7.00000000000000
> > > > >    6.00000000000000        7.00000000000000        8.00000000000000
> > > > >    7.00000000000000        8.00000000000000        9.00000000000000
> > > > >    8.00000000000000        9.00000000000000        10.0000000000000
> > > > >    9.00000000000000        10.0000000000000        1000.00000000000
> > > > >    10.0000000000000        1000.00000000000        100000.000000000
> > > > >
> > > > > First column if ok (1st component), but the other two columns show wrong numbers. For component 2 and 3, only the very last value is ok. It seems values from component b are copied from a, and those from component c from b, with an offset.
> > > > >
> > > > > What do you think ?
> > > > >
> > > > > Christophe
> > > > >
> > > > >
> > > > > On Fri, Sep 27, 2013 at 2:47 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > >
> > > > >   Indices start in fortran arrays from 1 by default so don't even try zero
> > > > >
> > > > >    I've attached a new version that has three components,
> > > > >  30 TOTAL vector entries and 10 for each component. Please run it and send the results back.
> > > > >
> > > > >     Barry
> > > > >
> > > > > On Sep 27, 2013, at 5:40 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> > > > >
> > > > > > Hi Barry,
> > > > > >
> > > > > > I made some tests with your code that allows to use struct in Fortran90. It compiles and does not complain with the struct with 2 components but...it exhibits some strange behaviour. Here are the behaviours I observed. To make it simple, I used n=10.
> > > > > >
> > > > > > - After using call VecSet(x,one,ierr), I checked the values of each component of the vector using
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >       do i = 0 , n
> > > > > >        write(*,*) i , xarray(i)%a, xarray(i)%b
> > > > > >       end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > > I checked the values from i= 0 to n since I was not sure what was the convention (0- or 1-based). I obtained:
> > > > > >
> > > > > >            0  0.000000000000000E+000   1.00000000000000
> > > > > >            1   1.00000000000000        1.00000000000000
> > > > > >            2   1.00000000000000        1.00000000000000
> > > > > >            3   1.00000000000000        1.00000000000000
> > > > > >            4   1.00000000000000        1.00000000000000
> > > > > >            5   1.00000000000000        1.00000000000000
> > > > > >            6   1.00000000000000        1.00000000000000
> > > > > >            7   1.00000000000000        1.00000000000000
> > > > > >            8   1.00000000000000        1.00000000000000
> > > > > >            9   1.00000000000000        1.00000000000000
> > > > > >           10   1.00000000000000       1.996650376645798E-314
> > > > > >
> > > > > >
> > > > > > As you can see, for component a, there is a 0 for i=0, then the values are correct. For component b, correct values go from i=1 to 9.There is garbage in second component for i=10 . I checked that if you define a third component c, its values go from i=-1 to n-2. It's like if values of second component started at the end of first component, and same thing for third component.
> > > > > >
> > > > > > Then, instead of using VecSet(), I tried to assign values directly to each component of the vector with
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >       do i = 0 , n-1
> > > > > >        xarray(i)%a = 2.0
> > > > > >        xarray(i)%b = 3.0
> > > > > >       end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > > Here I checked that indices actually start at 0 and not at 1. With a loop from i=1 to n I got a Memory corruption message.
> > > > > >
> > > > > > and I checked that the values are correctly assigned with
> > > > > >
> > > > > >       call VecGetArrayMyStruct(x,xarray,ierr)
> > > > > >        do i = 0 , n-1
> > > > > >         write(*,*) i, xarray(i)%a, xarray(i)%b
> > > > > >        end do
> > > > > >       call VecRestoreArrayMyStruct(x,xarray,ierr)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Then, I obtained:
> > > > > >
> > > > > >            0   2.00000000000000        2.00000000000000
> > > > > >            1   2.00000000000000        2.00000000000000
> > > > > >            2   2.00000000000000        2.00000000000000
> > > > > >            3   2.00000000000000        2.00000000000000
> > > > > >            4   2.00000000000000        2.00000000000000
> > > > > >            5   2.00000000000000        2.00000000000000
> > > > > >            6   2.00000000000000        2.00000000000000
> > > > > >            7   2.00000000000000        2.00000000000000
> > > > > >            8   2.00000000000000        2.00000000000000
> > > > > >            9   2.00000000000000        3.00000000000000
> > > > > >
> > > > > >
> > > > > > Here the problem seems more severe. Values are not correctly assigned to the second component. There should be 3.0 in the second column. It seems values of the first component are copied to the second one. Only for i=10 the value of xarray(i)%b is correct (3.0).
> > > > > >
> > > > > > Any idea where it could come from ?
> > > > > > I guess it needs some fixes here and there but I think your idea is good and that it could work.
> > > > > >
> > > > > > Christophe
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 26, 2013 at 5:45 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > >
> > > > > >    I just put it with the Fortran source code and compile it with the rest of the application code; here is the makefile I used
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sep 26, 2013, at 10:14 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> > > > > >
> > > > > > > By the way, what should I do with the small .c code ? Where should I put it ?
> > > > > > >
> > > > > > > Christophe
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 26, 2013 at 4:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > > > > >
> > > > > > >    Christophe,
> > > > > > >
> > > > > > >      Despite my LinkIn Endorsement for expertise in Fortran :-) I cannot pretend to be an expert in FortranXX but I have cooked up an example demonstrating accessing the Vec entries as if they are in an array of derived types. I've attached the example code; there needs to be a small C stub that defines the functions for your specific derived type name.
> > > > > > > Note that it will only work I think if your N is a compile time constant.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >     It worked with
> > > > > > > ~/Downloads$ gfortran --version
> > > > > > > GNU Fortran (GCC) 4.8.1 20130404 (prerelease)
> > > > > > >
> > > > > > >
> > > > > > >     I do not understand exactly why it works since it uses F90Array1dCreate(fa,PETSC_SCALAR,1,len,ptr PETSC_F90_2PTR_PARAM(ptrd)); which has a single PETSC_SCALAR as a building block but ? I hope it works for you. If it doesn't, let us know the compiler you are using and we may be able to get it working for that compiler.
> > > > > > >
> > > > > > >    Barry
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sep 26, 2013, at 4:41 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Me again !
> > > > > > > >
> > > > > > > > I have read in previous posts that it is hard in Fortran to declare something similar to a typedef struct in C to manage a multicomponent problem.
> > > > > > > >
> > > > > > > > Is it still the case ? Has the problem been solved ?
> > > > > > > >
> > > > > > > > I am asking because my plan is to implement a multicomponent problem (in 1D), with many components that will be organized in arrays of two dimensions. In C I could define
> > > > > > > >
> > > > > > > > typedef struct{
> > > > > > > > PetscScalar U;
> > > > > > > > PetscScalar V;
> > > > > > > > PetscScalar A[N][N];
> > > > > > > > } Field;
> > > > > > > >
> > > > > > > > and then I could calculate the residual function with
> > > > > > > >
> > > > > > > > F[i].U = ...
> > > > > > > > F[i].V = ...
> > > > > > > > F[i].A[k][n] = ...
> > > > > > > >
> > > > > > > > which is quite convenient.
> > > > > > > >
> > > > > > > > If in Fortran it is not possible to use a struct as in C, I am afraid I'll have to deal with
> > > > > > > >
> > > > > > > > F(jdof,i) = ...
> > > > > > > >
> > > > > > > > where I will have only jdof to address U, V and A[ ][ ], which can be difficult and not very convenient I guess. Before I move all my code to C, does anyone have an alternative idea to manage this multi(many)component problem in Fortran ?
> > > > > > > >
> > > > > > > > Many thanks in advance for your help and suggestion !
> > > > > > > >
> > > > > > > > Christophe
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
> 
> 


From jedbrown at mcs.anl.gov  Tue Oct  1 07:01:02 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 07:01:02 -0500
Subject: [petsc-users] Back to struct in Fortran to represent field with
	dof > 1
In-Reply-To: <4B31F44E-CE2D-433D-9738-306446C30F95@mcs.anl.gov>
References: <CANBrw+52SARdE0Nr0RUh094nd=CQ=UkApr3sgHNvaqihjdqT7w@mail.gmail.com>
	<B395E1EA-628A-417D-94EC-5A62C653C916@mcs.anl.gov>
	<CANBrw+7O+8MZQ1F1ZbJeF8CZAQeSFUCsusYHbPvo--5JzAV+pQ@mail.gmail.com>
	<CANBrw+7=1N=Cd14x30MppriShh-_NypuV2SLbj4aUe_ApB6bVA@mail.gmail.com>
	<3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov>
	<CANBrw+4ugcmFORuZhDy2sCG86P_GxyePa3u8CNMqNBvv6cnqxw@mail.gmail.com>
	<B77F0A05-850C-42B2-9451-614D732F940C@mcs.anl.gov>
	<CANBrw+6yXPgR8FU1v3qsqQYJJi6s+7ZSUiPaZuverR4sefy3jA@mail.gmail.com>
	<DED33F6A-35AB-4422-9612-6E3A32F8D6AC@mcs.anl.gov>
	<CANBrw+7tocNMkaFdyQ5xYVhepy-_7fmTPihK=8_ExGOVfJjYvA@mail.gmail.com>
	<A4273914-DCAB-4B2A-B0B7-B457898DDFE1@mcs.anl.gov>
	<CANBrw+49+1jcKmUtFHmP-50x_mecDmpnx4bfGWKDk9f0j04DxQ@mail.gmail.com>
	<33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov>
	<CANBrw+4uxe7oR7Du0MKEa=nNKQV8PqGq0HhO=K183m9ScH0Lrg@mail.gmail.com>
	<214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
	<CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>
	<4B31F44E-CE2D-433D-9738-306446C30F95@mcs.anl.gov>
Message-ID: <87d2np5hfl.fsf@mcs.anl.gov>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    I put the example with VecGetArray() into petsc-dev and if you send
>    me one with DMDAVecGetArray() I'll add that as well.  Since the
>    derived type has to appear explicitly in the code I can't put
>    something in the library only in examples that people can copy and
>    change.

This is the sad state of Fortran.  Alternatively, it should be possible
to call the C DMDAVecGetArray via the ISO C bindings in Fortran 2003.

Yet another example that the best way to call Fortran libraries from
Fortran is to use the ISO C bindings.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/075ae953/attachment.pgp>

From christophe.ortiz at ciemat.es  Tue Oct  1 07:32:20 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 1 Oct 2013 14:32:20 +0200
Subject: [petsc-users] Back to struct in Fortran to represent field with
 dof > 1
In-Reply-To: <87d2np5hfl.fsf@mcs.anl.gov>
References: <CANBrw+52SARdE0Nr0RUh094nd=CQ=UkApr3sgHNvaqihjdqT7w@mail.gmail.com>
	<B395E1EA-628A-417D-94EC-5A62C653C916@mcs.anl.gov>
	<CANBrw+7O+8MZQ1F1ZbJeF8CZAQeSFUCsusYHbPvo--5JzAV+pQ@mail.gmail.com>
	<CANBrw+7=1N=Cd14x30MppriShh-_NypuV2SLbj4aUe_ApB6bVA@mail.gmail.com>
	<3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov>
	<CANBrw+4ugcmFORuZhDy2sCG86P_GxyePa3u8CNMqNBvv6cnqxw@mail.gmail.com>
	<B77F0A05-850C-42B2-9451-614D732F940C@mcs.anl.gov>
	<CANBrw+6yXPgR8FU1v3qsqQYJJi6s+7ZSUiPaZuverR4sefy3jA@mail.gmail.com>
	<DED33F6A-35AB-4422-9612-6E3A32F8D6AC@mcs.anl.gov>
	<CANBrw+7tocNMkaFdyQ5xYVhepy-_7fmTPihK=8_ExGOVfJjYvA@mail.gmail.com>
	<A4273914-DCAB-4B2A-B0B7-B457898DDFE1@mcs.anl.gov>
	<CANBrw+49+1jcKmUtFHmP-50x_mecDmpnx4bfGWKDk9f0j04DxQ@mail.gmail.com>
	<33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov>
	<CANBrw+4uxe7oR7Du0MKEa=nNKQV8PqGq0HhO=K183m9ScH0Lrg@mail.gmail.com>
	<214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
	<CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>
	<4B31F44E-CE2D-433D-9738-306446C30F95@mcs.anl.gov>
	<87d2np5hfl.fsf@mcs.anl.gov>
Message-ID: <CANBrw+6VHt9rnw+f1AAgxJafj-s2hHc=Ox-Cyqm7+KcwVxpwXw@mail.gmail.com>

On Tue, Oct 1, 2013 at 2:01 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> >    I put the example with VecGetArray() into petsc-dev and if you send
> >    me one with DMDAVecGetArray() I'll add that as well.  Since the
> >    derived type has to appear explicitly in the code I can't put
> >    something in the library only in examples that people can copy and
> >    change.
>
> This is the sad state of Fortran.  Alternatively, it should be possible
> to call the C DMDAVecGetArray via the ISO C bindings in Fortran 2003.
>
>
I will check that. I have the book fortran 95/2003 by Metcalf and there is
a chapter dedicated to iso c bindings.


> Yet another example that the best way to call Fortran libraries from
> Fortran is to use the ISO C bindings.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/9d617e74/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct  1 08:05:53 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Oct 2013 08:05:53 -0500
Subject: [petsc-users] Back to struct in Fortran to represent field with
	dof > 1
In-Reply-To: <87d2np5hfl.fsf@mcs.anl.gov>
References: <CANBrw+52SARdE0Nr0RUh094nd=CQ=UkApr3sgHNvaqihjdqT7w@mail.gmail.com>
	<B395E1EA-628A-417D-94EC-5A62C653C916@mcs.anl.gov>
	<CANBrw+7O+8MZQ1F1ZbJeF8CZAQeSFUCsusYHbPvo--5JzAV+pQ@mail.gmail.com>
	<CANBrw+7=1N=Cd14x30MppriShh-_NypuV2SLbj4aUe_ApB6bVA@mail.gmail.com>
	<3524B3B9-C5FE-46A7-8556-F36DEEC287BD@mcs.anl.gov>
	<CANBrw+4ugcmFORuZhDy2sCG86P_GxyePa3u8CNMqNBvv6cnqxw@mail.gmail.com>
	<B77F0A05-850C-42B2-9451-614D732F940C@mcs.anl.gov>
	<CANBrw+6yXPgR8FU1v3qsqQYJJi6s+7ZSUiPaZuverR4sefy3jA@mail.gmail.com>
	<DED33F6A-35AB-4422-9612-6E3A32F8D6AC@mcs.anl.gov>
	<CANBrw+7tocNMkaFdyQ5xYVhepy-_7fmTPihK=8_ExGOVfJjYvA@mail.gmail.com>
	<A4273914-DCAB-4B2A-B0B7-B457898DDFE1@mcs.anl.gov>
	<CANBrw+49+1jcKmUtFHmP-50x_mecDmpnx4bfGWKDk9f0j04DxQ@mail.gmail.com>
	<33F8252E-1F7C-487E-9EFC-83CC60852471@mcs.anl.gov>
	<CANBrw+4uxe7oR7Du0MKEa=nNKQV8PqGq0HhO=K183m9ScH0Lrg@mail.gmail.com>
	<214E26C8-54A1-4E4B-B7A4-5EBBC4E50B40@mcs.anl.gov>
	<CANBrw+4gpoRqMtNHK+hSo7JC6Dvd_H5tnUp0jQAS1Yd8KXe37g@mail.gmail.com>
	<4B31F44E-CE2D-433D-9738-306446C30F95@mcs.anl.gov>
	<87d2np5hfl.fsf@mcs.anl.gov>
Message-ID: <05D02502-F190-4CF6-B771-0CFAC12F71EC@mcs.anl.gov>



   I doubt very much that the ISO C bindings allow one to pass a Fortran pointer through a C function which is what we need to do so I'd be surprised if the ISO C bindings will save the day. But you are welcome to try.

    Barry

On Oct 1, 2013, at 7:01 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>   I put the example with VecGetArray() into petsc-dev and if you send
>>   me one with DMDAVecGetArray() I'll add that as well.  Since the
>>   derived type has to appear explicitly in the code I can't put
>>   something in the library only in examples that people can copy and
>>   change.
> 
> This is the sad state of Fortran.  Alternatively, it should be possible
> to call the C DMDAVecGetArray via the ISO C bindings in Fortran 2003.
> 
> Yet another example that the best way to call Fortran libraries from
> Fortran is to use the ISO C bindings.


From popov at uni-mainz.de  Tue Oct  1 11:35:27 2013
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 1 Oct 2013 18:35:27 +0200
Subject: [petsc-users] MatCreateSchurComplement
In-Reply-To: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu>
References: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu>
Message-ID: <524AF9CF.4090905@uni-mainz.de>

Dear all,

I guess MatCreateSchurComplement is somehow broken in PETSc 3.4.2

Attached is the modified ex23 from src/ksp/ksp/examples/tutorials

I just copied system matrix three times and created a Schur complement 
object.

However mpirun -n 4 ./ex23 gives: Unknown Mat type given: schurcomplement!

[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: MatSetType() line 63 in 
/Users/anton/LIB/petsc-3.4.2/src/mat/interface/matreg.c
[0]PETSC ERROR: MatCreateSchurComplement() line 180 in 
/Users/anton/LIB/petsc-3.4.2/src/ksp/ksp/utils/schurm.c
[0]PETSC ERROR: main() line 121 in src/ksp/ksp/examples/tutorials/ex23.c
[1]PETSC ERROR: --------------------- Error Message 
------------------------------------
[1]PETSC ERROR: Unknown type. Check for miss-spelling or missing 
external package needed for type:
see http://www.mcs.anl.gov/petsc/documentation/installation.html#external!
[1]PETSC ERROR: Unknown Mat type given: schurcomplement!
[1]PETSC ERROR: 
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: 
------------------------------------------------------------------------
[1]PETSC ERROR: ./ex23 on a arch-darwin-c-debug named 
mac11-005.geo.uni-mainz.de by anton Tue Oct  1 18:30:50 2013
[1]PETSC ERROR: Libraries linked from /opt/petsc/petsc-3.4.2-int32-debug/lib
[1]PETSC ERROR: Configure run at Mon Sep 30 12:17:59 2013
[1]PETSC ERROR: Configure options 
--prefix=/opt/petsc/petsc-3.4.2-int32-debug --download-f-blas-lapack=1 
--with-debugging=1 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind 
--COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" 
--with-shared-libraries=0 --with-cc=mpicc --with-cxx=mpicxx 
--with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 
--download-scalapack=1 --download-metis=1 --download-parmetis=1 
--download-mumps=1 --download-superlu_dist=1

Similar error we have in our code if we use MatCreateSchurComplement 
function explicitely.

Any idea?

Thanks,

Anton
-------------- next part --------------

static char help[] = "Solves a tridiagonal linear system.\n\n";

/*T
   Concepts: KSP^basic parallel example;
   Processors: n
T*/

/*
  Include "petscksp.h" so that we can use KSP solvers.  Note that this file
  automatically includes:
     petscsys.h       - base PETSc routines   petscvec.h - vectors
     petscmat.h - matrices
     petscis.h     - index sets            petscksp.h - Krylov subspace methods
     petscviewer.h - viewers               petscpc.h  - preconditioners

  Note:  The corresponding uniprocessor example is ex1.c
*/
#include <petscksp.h>

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **args)
{
  Vec            x, b, u;          /* approx solution, RHS, exact solution */
  Mat            A;                /* linear system matrix */
  Mat            L, D, U, S;
  KSP            ksp;              /* linear solver context */
  PC             pc;               /* preconditioner context */
  PetscReal      norm,tol=1.e-11;  /* norm of solution error */
  PetscErrorCode ierr;
  PetscInt       i,n = 10,col[3],its,rstart,rend,nlocal;
  PetscScalar    neg_one = -1.0,one = 1.0,value[3];

  PetscInitialize(&argc,&args,(char*)0,help);
  ierr = PetscOptionsGetInt(NULL,"-n",&n,NULL);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
         Compute the matrix and right-hand-side vector that define
         the linear system, Ax = b.
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  /*
     Create vectors.  Note that we form 1 vector from scratch and
     then duplicate as needed. For this simple case let PETSc decide how
     many elements of the vector are stored on each processor. The second
     argument to VecSetSizes() below causes PETSc to decide.
  */
  ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr);
  ierr = VecSetSizes(x,PETSC_DECIDE,n);CHKERRQ(ierr);
  ierr = VecSetFromOptions(x);CHKERRQ(ierr);
  ierr = VecDuplicate(x,&b);CHKERRQ(ierr);
  ierr = VecDuplicate(x,&u);CHKERRQ(ierr);

  /* Identify the starting and ending mesh points on each
     processor for the interior part of the mesh. We let PETSc decide
     above. */

  ierr = VecGetOwnershipRange(x,&rstart,&rend);CHKERRQ(ierr);
  ierr = VecGetLocalSize(x,&nlocal);CHKERRQ(ierr);

  /*
     Create matrix.  When using MatCreate(), the matrix format can
     be specified at runtime.

     Performance tuning note:  For problems of substantial size,
     preallocation of matrix memory is crucial for attaining good
     performance. See the matrix chapter of the users manual for details.

     We pass in nlocal as the "local" size of the matrix to force it
     to have the same parallel layout as the vector created above.
  */
  ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
  ierr = MatSetSizes(A,nlocal,nlocal,n,n);CHKERRQ(ierr);
  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
  ierr = MatSetUp(A);CHKERRQ(ierr);

  /*
     Assemble matrix.

     The linear system is distributed across the processors by
     chunks of contiguous rows, which correspond to contiguous
     sections of the mesh on which the problem is discretized.
     For matrix assembly, each processor contributes entries for
     the part that it owns locally.
  */


  if (!rstart) {
    rstart = 1;
    i      = 0; col[0] = 0; col[1] = 1; value[0] = 2.0; value[1] = -1.0;
    ierr   = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr);
  }
  if (rend == n) {
    rend = n-1;
    i    = n-1; col[0] = n-2; col[1] = n-1; value[0] = -1.0; value[1] = 2.0;
    ierr = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr);
  }

  /* Set entries corresponding to the mesh interior */
  value[0] = -1.0; value[1] = 2.0; value[2] = -1.0;
  for (i=rstart; i<rend; i++) {
    col[0] = i-1; col[1] = i; col[2] = i+1;
    ierr   = MatSetValues(A,1,&i,3,col,value,INSERT_VALUES);CHKERRQ(ierr);
  }

  /* Assemble the matrix */
  ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);





	// duplicate matrix for test purposes
	ierr = MatDuplicate(A, MAT_COPY_VALUES, &L); CHKERRQ(ierr);
	ierr = MatDuplicate(A, MAT_COPY_VALUES, &D); CHKERRQ(ierr);
	ierr = MatDuplicate(A, MAT_COPY_VALUES, &U); CHKERRQ(ierr);

	// just check Schur complement creation
	ierr = MatCreateSchurComplement(A, A, L, U, D, &S); CHKERRQ(ierr);

	// destroy test matrices
	ierr = MatDestroy(&L); CHKERRQ(ierr);
	ierr = MatDestroy(&D); CHKERRQ(ierr);
	ierr = MatDestroy(&U); CHKERRQ(ierr);
	ierr = MatDestroy(&S); CHKERRQ(ierr);





  /*
     Set exact solution; then compute right-hand-side vector.
  */
  ierr = VecSet(u,one);CHKERRQ(ierr);
  ierr = MatMult(A,u,b);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                Create the linear solver and set various options
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  /*
     Create linear solver context
  */
  ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);

  /*
     Set operators. Here the matrix that defines the linear system
     also serves as the preconditioning matrix.
  */
  ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

  /*
     Set linear solver defaults for this problem (optional).
     - By extracting the KSP and PC contexts from the KSP context,
       we can then directly call any KSP and PC routines to set
       various options.
     - The following four statements are optional; all of these
       parameters could alternatively be specified at runtime via
       KSPSetFromOptions();
  */
  ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);
  ierr = PCSetType(pc,PCJACOBI);CHKERRQ(ierr);
  ierr = KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);

  /*
    Set runtime options, e.g.,
        -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
    These options will override those specified above as long as
    KSPSetFromOptions() is called _after_ any other customization
    routines.
  */
  ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                      Solve the linear system
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  /*
     Solve linear system
  */
  ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr);

  /*
     View solver info; we could instead use the option -ksp_view to
     print this info to the screen at the conclusion of KSPSolve().
  */
  ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                      Check solution and clean up
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  /*
     Check the error
  */
  ierr = VecAXPY(x,neg_one,u);CHKERRQ(ierr);
  ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr);
  ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);
  if (norm > tol) {
    ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %G, Iterations %D\n",norm,its);CHKERRQ(ierr);
  }

  /*
     Free work space.  All PETSc objects should be destroyed when they
     are no longer needed.
  */
  ierr = VecDestroy(&x);CHKERRQ(ierr); ierr = VecDestroy(&u);CHKERRQ(ierr);
  ierr = VecDestroy(&b);CHKERRQ(ierr); ierr = MatDestroy(&A);CHKERRQ(ierr);
  ierr = KSPDestroy(&ksp);CHKERRQ(ierr);

  /*
     Always call PetscFinalize() before exiting a program.  This routine
       - finalizes the PETSc libraries as well as MPI
       - provides summary and diagnostic information if certain runtime
         options are chosen (e.g., -log_summary).
  */
  ierr = PetscFinalize();
  return 0;
}

From jedbrown at mcs.anl.gov  Tue Oct  1 14:45:51 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 14:45:51 -0500
Subject: [petsc-users] MatCreateSchurComplement
In-Reply-To: <524AF9CF.4090905@uni-mainz.de>
References: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu>
	<524AF9CF.4090905@uni-mainz.de>
Message-ID: <877gdw4vww.fsf@mcs.anl.gov>

Anton Popov <popov at uni-mainz.de> writes:

> Dear all,
>
> I guess MatCreateSchurComplement is somehow broken in PETSc 3.4.2

You can fix it by calling KSPMatRegisterAll() before
MatCreateSchurComplement, and the issue is now fixed in 'maint' (it was
accidentally only fixed in 'master').

This problem came from simplifying dynamic loading and when the
backward-dependency was removed, we still considered KSPMatRegisterAll
to be collective on PETSC_COMM_WORLD, so it couldn't safely be called by
MatCreateSchurComplement.  Looks like you are the "rare" case below.


commit f550243c612070fecd53ff91e1d8ee65a2c6e25c
Author: Jed Brown <jed at 59A2.org>
Date:   Wed Oct 17 16:16:08 2012 -0500

    Add KSPMatRegisterAll() to avoid dependency of libpetscmat on libpetscksp
    
    Register MATSCHURCOMPLEMENT there. User should explicitly call
    KSPMatRegisterAll() or KSPInitializePackage() if they want to make a
    MATSCHURCOMPLEMENT before calling KSPCreate. Such a use case should be
    very rare.
    
    Hg-commit: 3c58006af47185551c8aeaf6e1c32e68b79916fb

 include/petscksp.h                 |  1 +
 src/ksp/ksp/interface/dlregisksp.c |  2 ++
 src/ksp/ksp/utils/schurm.c         | 25 +++++++++++++++++++++++++
 src/mat/interface/matregis.c       |  2 --
 4 files changed, 28 insertions(+), 2 deletions(-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/3bda2bc3/attachment.pgp>

From popov at uni-mainz.de  Tue Oct  1 15:36:40 2013
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 1 Oct 2013 22:36:40 +0200
Subject: [petsc-users] MatCreateSchurComplement
In-Reply-To: <877gdw4vww.fsf@mcs.anl.gov>
References: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu>
	<524AF9CF.4090905@uni-mainz.de> <877gdw4vww.fsf@mcs.anl.gov>
Message-ID: <524B3258.3030903@uni-mainz.de>

On 10/1/13 9:45 PM, Jed Brown wrote:
> Anton Popov <popov at uni-mainz.de> writes:
>
>> Dear all,
>>
>> I guess MatCreateSchurComplement is somehow broken in PETSc 3.4.2
> You can fix it by calling KSPMatRegisterAll() before
> MatCreateSchurComplement,
Thanks, Jed. It solves the problem.
>   and the issue is now fixed in 'maint' (it was
> accidentally only fixed in 'master').
>
> This problem came from simplifying dynamic loading and when the
> backward-dependency was removed, we still considered KSPMatRegisterAll
> to be collective on PETSC_COMM_WORLD, so it couldn't safely be called by
> MatCreateSchurComplement.  Looks like you are the "rare" case below.
>
>
> commit f550243c612070fecd53ff91e1d8ee65a2c6e25c
> Author: Jed Brown <jed at 59A2.org>
> Date:   Wed Oct 17 16:16:08 2012 -0500
>
>      Add KSPMatRegisterAll() to avoid dependency of libpetscmat on libpetscksp
>      
>      Register MATSCHURCOMPLEMENT there. User should explicitly call
>      KSPMatRegisterAll() or KSPInitializePackage() if they want to make a
>      MATSCHURCOMPLEMENT before calling KSPCreate. Such a use case should be
>      very rare.
>      
>      Hg-commit: 3c58006af47185551c8aeaf6e1c32e68b79916fb
>
>   include/petscksp.h                 |  1 +
>   src/ksp/ksp/interface/dlregisksp.c |  2 ++
>   src/ksp/ksp/utils/schurm.c         | 25 +++++++++++++++++++++++++
>   src/mat/interface/matregis.c       |  2 --
>   4 files changed, 28 insertions(+), 2 deletions(-)


From choi240 at purdue.edu  Tue Oct  1 16:00:14 2013
From: choi240 at purdue.edu (Joon Hee Choi)
Date: Tue, 1 Oct 2013 17:00:14 -0400 (EDT)
Subject: [petsc-users] The product of two MPIDENSE matrices
In-Reply-To: <85419958.159902.1380594346277.JavaMail.root@mailhub028.itcs.purdue.edu>
Message-ID: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>

Dear Petsc Developers,

Hello. 
I am wondering whether I can multiply MATELEMENTAL and MATAIJ matrices. If I cannot multiply those, could you build the multiplication?
Thank you.

Joon



----- Original Message -----
From: "Joon Hee Choi" <choi240 at purdue.edu>
To: "Jack Poulson" <poulson at stanford.edu>
Cc: "PETSc users list" <petsc-users at mcs.anl.gov>, "dev" <dev at libelemental.org>, "Jed Brown" <jedbrown at mcs.anl.gov>, "Hong Zhang" <hzhang at mcs.anl.gov>, "S V N Vishwanathan" <vishy at stat.purdue.edu>
Sent: Monday, September 30, 2013 10:25:46 PM
Subject: Re: [petsc-users] The product of two MPIDENSE matrices

> Dear all,

> I tried to compile elemental with 64 bit support by modifying the PETSc build script (see attached elemental.py). I modified the typedefs in Elemental and pointed the build script to a location on my  > home page. I also needed to add -std=c++11 to the CMake arguments lists. This allows Elemental to be built without errors. However, the PETSc build fails with a strange configuration error, which seems > to be related to C++11. Note that I have included --CXXFLAGS="-std=c++11" in the configure options to PETSc.

> The configure.log file can be downloaded from

> http://www.stat.purdue.edu/~vishy/configure.log

> any advice on how to workaround this issue is greatly appreciated.

> thanks


----- ?? ??? -----
?? ??: "Jack Poulson" <poulson at stanford.edu>
?? ??: "Jed Brown" <jedbrown at mcs.anl.gov>, "Joon Hee Choi" <choi240 at purdue.edu>, "Hong Zhang" <hzhang at mcs.anl.gov>
??: "PETSc users list" <petsc-users at mcs.anl.gov>, "<dev at libelemental.org>" <dev at libelemental.org>
?? ??: 2013? 9? 30?, ??? ?? 7:59:22
??: Re: [petsc-users] The product of two MPIDENSE matrices

On 09/30/2013 12:26 PM, Jed Brown wrote:
>>>> Jack Poulson <poulson at stanford.edu> writes:
>>>> It hasn't yet been rigorously tested, but support for 64-bit
>>>> integers should be possible by simply changing the typedefs for
>>>> elem::Int and elem::Unsigned from 'int' -> 'long long int' and
>>>> 'unsigned' to 'long long unsigned'. These are guaranteed to be
>>>> 64-bit by C++11.
>>>> 
>>> Can you add a CMake configure option for these?  I'd rather 
>>> --download-elemental use an interface other than stream-editing
>>> headers.
>>> 

>> Done:
>> https://github.com/elemental/Elemental/commit/03d864e2a8ae01877194bbb572144568454f7d05
>> Configuring Elemental with the "-D USE_64BIT_INTS=ON" option will now
>> use 'long long' integers.

>> Also, the preceding commit essentially finished the Bunch-Kaufman support:
>> https://github.com/elemental/Elemental/commit/d2c79c6cfddab6c2ad9c84c49a0ee23b8fe27893

>> Jack

From jedbrown at mcs.anl.gov  Tue Oct  1 16:27:50 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 16:27:50 -0500
Subject: [petsc-users] The product of two MPIDENSE matrices
In-Reply-To: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>
References: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>
Message-ID: <87siwk3cmh.fsf@mcs.anl.gov>

Joon Hee Choi <choi240 at purdue.edu> writes:

> Dear Petsc Developers,
>
> Hello. 
> I am wondering whether I can multiply MATELEMENTAL and MATAIJ matrices. If I cannot multiply those, could you build the multiplication?

What are the dimensions of the matrices?

Jack, does Clique support multiplication of sparse matrices with [MC,MR]
dense matrices?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/90a8022b/attachment.pgp>

From wumengda at gmail.com  Tue Oct  1 16:49:29 2013
From: wumengda at gmail.com (Mengda Wu)
Date: Tue, 1 Oct 2013 17:49:29 -0400
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
Message-ID: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>

Hello all,

   I am trying to compile 64 bit version of petsc-3.4.2 using MinGW-w64
(latest version 4.8.1 installed with
http://sourceforge.net/projects/mingwbuilds/files/mingw-builds-install/mingw-builds-install.exe/download)
on Windows 7 64 bit. I have cygwin, python (under cygwin) installed.

  I run the "$ ./config/configure.py --with-cc='gcc' --with-fc='gfortran'
--with-cxx='g++'
--with-clanguage=cxx  --with-mpi=0 --with-debugging=0  --with-sowing=0
--with-c
2html=0". I got some errors in checkCCompiler. The log file is as follows:
============================================================
TEST checkCCompiler from
config.setCompilers(/cygdrive/c/Library/PETSc/download/petsc-3.4.2/config/BuildSystem/config/setCompilers.py:525)
TESTING: checkCCompiler from
config.setCompilers(config/BuildSystem/config/setCompilers.py:525)
  Locate a functional C compiler
Checking for program /home/glxiong/bin/gcc...not found
Checking for program /usr/local/bin/gcc...not found
Checking for program /usr/bin/gcc...not found
Checking for program
/cygdrive/c/Library/MinGW/x64-4.8.1/mingw64/bin/gcc...found
          Defined make macro "CC" to "gcc"
          Pushing language C
                    All intermediate test results are stored in
/tmp/petsc-novY5r/config.setCompilers
sh: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
-I/tmp/petsc-novY5r/config.setCompilers
/tmp/petsc-novY5r/config.setCompilers/conftest.c
Executing: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
-I/tmp/petsc-novY5r/config.setCompilers
/tmp/petsc-novY5r/config.setCompilers/conftest.c
sh:
Possible ERROR while running compiler: ret = 256
error message = {gcc.exe: error:
/tmp/petsc-novY5r/config.setCompilers/conftest.c: No such file or directory

gcc.exe: fatal error: no input files

compilation terminated.
============================================================

I think the problem is the MinGW-w64 compiler does not recognize the cygwin
path like /tmp. Is there a workaround? I tried to use a native Windows
python, but the configure fails to find the "make" command then. I guess
MinGW may be not supported by PetSc yet.

However, I found someone has successfully compiled a MPI-version Petsc on
Windows 64 bit. Here is the link:
http://geuz.org/pipermail/gmsh/2012/007589.html.
How is that possible?


Thanks,
Mengda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/394b973f/attachment.html>

From balay at mcs.anl.gov  Tue Oct  1 16:55:10 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 1 Oct 2013 16:55:10 -0500 (CDT)
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>

Looks like this mingw install is not from cygwin - and does not work with cygwin paths.

Suggest using mingw from cygwin - or install and use cygwin64 [which has non-mingw gcc-64]
You might need the following patch
https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207

Satish


On Tue, 1 Oct 2013, Mengda Wu wrote:

> Hello all,
> 
>    I am trying to compile 64 bit version of petsc-3.4.2 using MinGW-w64
> (latest version 4.8.1 installed with
> http://sourceforge.net/projects/mingwbuilds/files/mingw-builds-install/mingw-builds-install.exe/download)
> on Windows 7 64 bit. I have cygwin, python (under cygwin) installed.
> 
>   I run the "$ ./config/configure.py --with-cc='gcc' --with-fc='gfortran'
> --with-cxx='g++'
> --with-clanguage=cxx  --with-mpi=0 --with-debugging=0  --with-sowing=0
> --with-c
> 2html=0". I got some errors in checkCCompiler. The log file is as follows:
> ============================================================
> TEST checkCCompiler from
> config.setCompilers(/cygdrive/c/Library/PETSc/download/petsc-3.4.2/config/BuildSystem/config/setCompilers.py:525)
> TESTING: checkCCompiler from
> config.setCompilers(config/BuildSystem/config/setCompilers.py:525)
>   Locate a functional C compiler
> Checking for program /home/glxiong/bin/gcc...not found
> Checking for program /usr/local/bin/gcc...not found
> Checking for program /usr/bin/gcc...not found
> Checking for program
> /cygdrive/c/Library/MinGW/x64-4.8.1/mingw64/bin/gcc...found
>           Defined make macro "CC" to "gcc"
>           Pushing language C
>                     All intermediate test results are stored in
> /tmp/petsc-novY5r/config.setCompilers
> sh: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> -I/tmp/petsc-novY5r/config.setCompilers
> /tmp/petsc-novY5r/config.setCompilers/conftest.c
> Executing: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> -I/tmp/petsc-novY5r/config.setCompilers
> /tmp/petsc-novY5r/config.setCompilers/conftest.c
> sh:
> Possible ERROR while running compiler: ret = 256
> error message = {gcc.exe: error:
> /tmp/petsc-novY5r/config.setCompilers/conftest.c: No such file or directory
> 
> gcc.exe: fatal error: no input files
> 
> compilation terminated.
> ============================================================
> 
> I think the problem is the MinGW-w64 compiler does not recognize the cygwin
> path like /tmp. Is there a workaround? I tried to use a native Windows
> python, but the configure fails to find the "make" command then. I guess
> MinGW may be not supported by PetSc yet.
> 
> However, I found someone has successfully compiled a MPI-version Petsc on
> Windows 64 bit. Here is the link:
> http://geuz.org/pipermail/gmsh/2012/007589.html.
> How is that possible?
> 
> 
> Thanks,
> Mengda
> 


From vishy at stat.purdue.edu  Tue Oct  1 17:04:27 2013
From: vishy at stat.purdue.edu (S V N Vishwanathan)
Date: Tue, 01 Oct 2013 18:04:27 -0400
Subject: [petsc-users] The product of two MPIDENSE matrices
In-Reply-To: <87siwk3cmh.fsf@mcs.anl.gov>
References: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>
	<87siwk3cmh.fsf@mcs.anl.gov>
Message-ID: <m2vc1gk5qs.wl%vishy@stat.purdue.edu>

Hi

> > I am wondering whether I can multiply MATELEMENTAL and MATAIJ
> > matrices. If I cannot multiply those, could you build the
> > multiplication?
> 
> What are the dimensions of the matrices?
> 
> Jack, does Clique support multiplication of sparse matrices with
> [MC,MR] dense matrices?

The dense matrix is very flat and long (e.g. 10 x few 1000s) and the
sparse matrix is also flat and very very long (e.g. few 1000s x few
millions). I saw the code which is used to multiply a MPIAIJ matrix with
a dense matrix. We need something very similar, but which uses elemental
matrices. 

There seems to be some inconsistency right now in PETSc. I can multiple
a MPIAIJ matrix with MPIDENSE, but I cannot multiply MPIDENSE with
MPIDENSE since there is no PLAPACK support anymore. On other hand, I can
multiply a MATELEMENTAL with MATELEMENTAL but I cannot multiply a
MPIDENSE with MATELEMENTAL. Is the eventual plan to remove MPIDENSE
completely and replace it with MATELEMENTAL?

thanks

vishy

From balay at mcs.anl.gov  Tue Oct  1 17:04:42 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 1 Oct 2013 17:04:42 -0500 (CDT)
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
	<alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
Message-ID: <alpine.LFD.2.03.1310011701560.1897@mcs.anl.gov>

mingw64 from cygwin[32] appears to work fine with /tmp.

Satish

--------
balay at msnehalem2 ~/junk
$ which x86_64-w64-mingw32-gcc.exe
/usr/bin/x86_64-w64-mingw32-gcc.exe

balay at msnehalem2 ~/junk
$ cygcheck -f /usr/bin/x86_64-w64-mingw32-gcc.exe
mingw64-x86_64-gcc-core-4.7.3-1

balay at msnehalem2 ~/junk
$ cp sizeof.c /tmp

balay at msnehalem2 ~/junk
$ x86_64-w64-mingw32-gcc.exe /tmp/sizeof.c -o sizeof.exe

balay at msnehalem2 ~/junk
$ file sizeof.exe
sizeof.exe: PE32+ executable (console) x86-64, for MS Windows

balay at msnehalem2 ~/junk
$ ./sizeof.exe
long double : 16
double      : 8
int         : 4
char        : 1
short       : 2
long        : 4
long long   : 8
int *       : 8
size_t      : 8

balay at msnehalem2 ~/junk
$

On Tue, 1 Oct 2013, Satish Balay wrote:

> Looks like this mingw install is not from cygwin - and does not work with cygwin paths.
> 
> Suggest using mingw from cygwin - or install and use cygwin64 [which has non-mingw gcc-64]
> You might need the following patch
> https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207
> 
> Satish
> 
> 
> On Tue, 1 Oct 2013, Mengda Wu wrote:
> 
> > Hello all,
> > 
> >    I am trying to compile 64 bit version of petsc-3.4.2 using MinGW-w64
> > (latest version 4.8.1 installed with
> > http://sourceforge.net/projects/mingwbuilds/files/mingw-builds-install/mingw-builds-install.exe/download)
> > on Windows 7 64 bit. I have cygwin, python (under cygwin) installed.
> > 
> >   I run the "$ ./config/configure.py --with-cc='gcc' --with-fc='gfortran'
> > --with-cxx='g++'
> > --with-clanguage=cxx  --with-mpi=0 --with-debugging=0  --with-sowing=0
> > --with-c
> > 2html=0". I got some errors in checkCCompiler. The log file is as follows:
> > ============================================================
> > TEST checkCCompiler from
> > config.setCompilers(/cygdrive/c/Library/PETSc/download/petsc-3.4.2/config/BuildSystem/config/setCompilers.py:525)
> > TESTING: checkCCompiler from
> > config.setCompilers(config/BuildSystem/config/setCompilers.py:525)
> >   Locate a functional C compiler
> > Checking for program /home/glxiong/bin/gcc...not found
> > Checking for program /usr/local/bin/gcc...not found
> > Checking for program /usr/bin/gcc...not found
> > Checking for program
> > /cygdrive/c/Library/MinGW/x64-4.8.1/mingw64/bin/gcc...found
> >           Defined make macro "CC" to "gcc"
> >           Pushing language C
> >                     All intermediate test results are stored in
> > /tmp/petsc-novY5r/config.setCompilers
> > sh: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> > -I/tmp/petsc-novY5r/config.setCompilers
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c
> > Executing: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> > -I/tmp/petsc-novY5r/config.setCompilers
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c
> > sh:
> > Possible ERROR while running compiler: ret = 256
> > error message = {gcc.exe: error:
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c: No such file or directory
> > 
> > gcc.exe: fatal error: no input files
> > 
> > compilation terminated.
> > ============================================================
> > 
> > I think the problem is the MinGW-w64 compiler does not recognize the cygwin
> > path like /tmp. Is there a workaround? I tried to use a native Windows
> > python, but the configure fails to find the "make" command then. I guess
> > MinGW may be not supported by PetSc yet.
> > 
> > However, I found someone has successfully compiled a MPI-version Petsc on
> > Windows 64 bit. Here is the link:
> > http://geuz.org/pipermail/gmsh/2012/007589.html.
> > How is that possible?
> > 
> > 
> > Thanks,
> > Mengda
> > 
> 
> 


From poulson at stanford.edu  Tue Oct  1 17:15:23 2013
From: poulson at stanford.edu (Jack Poulson)
Date: Tue, 01 Oct 2013 15:15:23 -0700
Subject: [petsc-users] [dev@libelemental.org] Re: The product of two
 MPIDENSE matrices
In-Reply-To: <0MdsgT-1VCVTL1Z7d-00QNHn@ml.kundenserver.de>
References: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>	<87siwk3cmh.fsf@mcs.anl.gov>
	<0MdsgT-1VCVTL1Z7d-00QNHn@ml.kundenserver.de>
Message-ID: <524B497B.4000802@stanford.edu>

On 10/01/2013 03:04 PM, S V N Vishwanathan wrote:
> Hi
> 
>>> I am wondering whether I can multiply MATELEMENTAL and MATAIJ
>>> matrices. If I cannot multiply those, could you build the
>>> multiplication?
>>
>> What are the dimensions of the matrices?
>>
>> Jack, does Clique support multiplication of sparse matrices with
>> [MC,MR] dense matrices?
> 
> The dense matrix is very flat and long (e.g. 10 x few 1000s) and the
> sparse matrix is also flat and very very long (e.g. few 1000s x few
> millions). I saw the code which is used to multiply a MPIAIJ matrix with
> a dense matrix. We need something very similar, but which uses elemental
> matrices. 
> 

I haven't yet dedicated much time to the sparse matrix-vector
multiplication (my focus has been on the sparse-direct factorizations
and triangular solves), but there *is* a routine for forming

  Y := alpha A X + Y,

where A is a distributed sparse matrix, and Y has each column
identically distributed over the entire team of processes. This is ideal
for when Y consists of a few hundred or less vectors.

You seem to be describing a multiplication of the form

D2 := D1 S,

where D1 and D2 are very short dense matrices, and S is sparse. If you
set A := S', X := D1', and D2 := Y', then your use case is appropriate
for the above scheme. The implementation (and prototype) is here:

https://github.com/poulson/Clique/blob/11cdd5cf95a572a5289375873cb2a20d2d141810/include/clique/numeric/multiply.hpp#L16

Jack

From mirzadeh at gmail.com  Tue Oct  1 17:55:38 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 1 Oct 2013 15:55:38 -0700
Subject: [petsc-users] Scatter error
Message-ID: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>

Hi guys,

I have two ghosted vectors fxx, fyy which should store derivatives of
a quantity f. They are both obtained via calling VecDuplicate on f. To
minimize communications when computing both fxx and fyy I do the
following:

// loop over boundary points and compute fxx and fyy

VecGhostUpdateBegin(fxx, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateBegin(fyy, INSERT_VALUES, SCATTER_FORWARD);

// compute fxx and fyy for internal points

VecGhostUpdateEnd(fxx, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateEnd(fyy, INSERT_VALUES, SCATTER_FORWARD);

However this does not work and I get the following error:
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Object is in wrong state!
[1]PETSC ERROR:  Scatter ctx already in use!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Development HG revision:
d0a314355399bb0c38e384fde4b1feb01b721896  HG Date: Sat Nov 03 17:18:52
2012 -0500
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by
mohammad Tue Oct  1 15:47:38 2013
[1]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[1]PETSC ERROR: Configure run at Sat Nov  3 15:26:57 2012
[1]PETSC ERROR: Configure options --download-blacs=1
--download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1
--download-metis=1 --download-mumps=1 --download-parmetis=1
--download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx
--with-mpi-dir=/usr/bin --with-pthreadclasses=1
PETSC_ARCH=arch-linux2-cxx-debug
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: VecScatterBegin() line 1546 in
/home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c
[1]PETSC ERROR: VecGhostUpdateBegin() line 235 in
/home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c
[1]PETSC ERROR: constructing node neighboring information ... done in
0.0314 secs. on process 0 [Note: only showing root's timings]
Computing derivatives the new way ...
dxx_and_dyy_central() line 659 in
"unknowndirectory/"../../src/my_p4est_node_neighbors.cpp
terminate called after throwing an instance of 'PETSc::Exception'
  what():  std::exception
[mohammad-Desktop:19589] *** Process received signal ***
[mohammad-Desktop:19589] Signal: Aborted (6)
[mohammad-Desktop:19589] Signal code:  (-6)
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR:  Scatter ctx already in use!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision:
d0a314355399bb0c38e384fde4b1feb01b721896  HG Date: Sat Nov 03 17:18:52
2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./derivatives on a arch-linu named mohammad-Desktop by
mohammad Tue Oct  1 15:47:38 2013
[0]PETSC ERROR: Libraries linked from
/home/mohammad/soft/petsc-dev/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Sat Nov  3 15:26:57 2012
[0]PETSC ERROR: Configure options --download-blacs=1
--download-f-blas-lapack=1 --download-hdf5=1 --download-hypre=1
--download-metis=1 --download-mumps=1 --download-parmetis=1
--download-scalapack=1 --download-superlu_dist=1 --with-clanguage=cxx
--with-mpi-dir=/usr/bin --with-pthreadclasses=1
PETSC_ARCH=arch-linux2-cxx-debug
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecScatterBegin() line 1546 in
/home/mohammad/soft/petsc-dev/src/vec/vec/utils/vscat.c
[0]PETSC ERROR: VecGhostUpdateBegin() line 235 in
/home/mohammad/soft/petsc-dev/src/vec/vec/impls/mpi/commonmpvec.c
[0]PETSC ERROR: dxx_and_dyy_central() line 659 in
"unknowndirectory/"../../src/my_p4est_node_neighbors.cpp
terminate called after throwing an instance of 'PETSc::Exception'
  what():  std::exception
[mohammad-Desktop:19588] *** Process received signal ***
[mohammad-Desktop:19588] Signal: Aborted (6)
[mohammad-Desktop:19588] Signal code:  (-6)
[mohammad-Desktop:19589] [ 0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f12290f4cb0]
[mohammad-Desktop:19589] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1228a60425]
[mohammad-Desktop:19589] [ 2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1228a63b8b]
[mohammad-Desktop:19589] [ 3]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)
[0x7f122988069d]
[mohammad-Desktop:19589] [ 4]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f122987e846]
[mohammad-Desktop:19589] [ 5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f122987e873]
[mohammad-Desktop:19589] [ 6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f122987e96e]
[mohammad-Desktop:19589] [ 7]
./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c)
[0x5bf83c]
[mohammad-Desktop:19589] [ 8]
./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950)
[0x505c30]
[mohammad-Desktop:19589] [ 9] ./derivatives(main+0xe89) [0x4e4678]
[mohammad-Desktop:19589] [10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
[0x7f1228a4b76d]
[mohammad-Desktop:19589] [11] ./derivatives() [0x4e2ab9]
[mohammad-Desktop:19589] *** End of error message ***
[mohammad-Desktop:19588] [ 0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f48f260bcb0]
[mohammad-Desktop:19588] [ 1]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f48f1f77425]
[mohammad-Desktop:19588] [ 2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f48f1f7ab8b]
[mohammad-Desktop:19588] [ 3]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d)
[0x7f48f2d9769d]
[mohammad-Desktop:19588] [ 4]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f48f2d95846]
[mohammad-Desktop:19588] [ 5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f48f2d95873]
[mohammad-Desktop:19588] [ 6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f48f2d9596e]
[mohammad-Desktop:19588] [ 7]
./derivatives(_Z10PetscErrorP19ompi_communicator_tiPKcS2_S2_i14PetscErrorTypeS2_z+0x53c)
[0x5bf83c]
[mohammad-Desktop:19588] [ 8]
./derivatives(_ZNK25my_p4est_node_neighbors_t19dxx_and_dyy_centralEP6_p_VecS1_S1_+0x950)
[0x505c30]
[mohammad-Desktop:19588] [ 9] ./derivatives(main+0xe89) [0x4e4678]
[mohammad-Desktop:19588] [10]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)
[0x7f48f1f6276d]
[mohammad-Desktop:19588] [11] ./derivatives() [0x4e2ab9]
[mohammad-Desktop:19588] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 19589 on node
mohammad-Desktop exited on signal 6 (Aborted).
--------------------------------------------------------------------------


I assume I'm getting this since they somehow share a VecScatter
internally because of VecDuplicate? If so how can I fix this?

Also as a side question, if I stored fxx and fyy inside a block
vector, would I gain any performance in terms of MPI traffic? What
about in 3D?

Thanks

From choi240 at purdue.edu  Tue Oct  1 18:09:02 2013
From: choi240 at purdue.edu (Joon Hee Choi)
Date: Tue, 1 Oct 2013 19:09:02 -0400 (EDT)
Subject: [petsc-users] [dev@libelemental.org] Re: The product of two
 MPIDENSE matrices
In-Reply-To: <524B497B.4000802@stanford.edu>
Message-ID: <390172426.163780.1380668942300.JavaMail.root@mailhub028.itcs.purdue.edu>



----- Original Message -----
From: "Jack Poulson" <poulson at stanford.edu>
To: dev at libelemental.org, "Jed Brown" <jedbrown at mcs.anl.gov>
Cc: "Joon Hee Choi" <choi240 at purdue.edu>, "Hong Zhang" <hzhang at mcs.anl.gov>, "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, October 1, 2013 6:15:23 PM
Subject: Re: [dev at libelemental.org] Re: [petsc-users] The product of two MPIDENSE matrices

On 10/01/2013 03:04 PM, S V N Vishwanathan wrote:
>> Hi
>> 
>>>> I am wondering whether I can multiply MATELEMENTAL and MATAIJ
>>>> matrices. If I cannot multiply those, could you build the
>>>> multiplication?
>>>
>>> What are the dimensions of the matrices?
>>>
>>> Jack, does Clique support multiplication of sparse matrices with
>>> [MC,MR] dense matrices?
>> 
>> The dense matrix is very flat and long (e.g. 10 x few 1000s) and the
>> sparse matrix is also flat and very very long (e.g. few 1000s x few
>> millions). I saw the code which is used to multiply a MPIAIJ matrix with
>> a dense matrix. We need something very similar, but which uses elemental
>> matrices. 
>> 

>I haven't yet dedicated much time to the sparse matrix-vector
>multiplication (my focus has been on the sparse-direct factorizations
>and triangular solves), but there *is* a routine for forming

>  Y := alpha A X + Y,

>where A is a distributed sparse matrix, and Y has each column
>identically distributed over the entire team of processes. This is ideal
>for when Y consists of a few hundred or less vectors.

>You seem to be describing a multiplication of the form

>D2 := D1 S,

>where D1 and D2 are very short dense matrices, and S is sparse. If you
>set A := S', X := D1', and D2 := Y', then your use case is appropriate
>for the above scheme. The implementation (and prototype) is here:

Yes, we have the similar multiplication as the form. 
BTW, can we use the multiplication function with the MPIAIJ matrix we created in Petsc? Usually, the parallel matrix is row distributed matrix.

>https://github.com/poulson/Clique/blob/11cdd5cf95a572a5289375873cb2a20d2d141810/include/clique/numeric/multiply.hpp#L16

>Jack

From jedbrown at mcs.anl.gov  Tue Oct  1 18:19:35 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 18:19:35 -0500
Subject: [petsc-users] Scatter error
In-Reply-To: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>
References: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>
Message-ID: <87a9is37g8.fsf@mcs.anl.gov>

Mohammad Mirzadeh <mirzadeh at gmail.com> writes:
> I assume I'm getting this since they somehow share a VecScatter
> internally because of VecDuplicate? If so how can I fix this?

Hmm, there is no VecScatterDuplicate so I think you'd have to make the
vectors independently.

> Also as a side question, if I stored fxx and fyy inside a block
> vector, would I gain any performance in terms of MPI traffic? What
> about in 3D?

It would be half the messages.  Why are these vectors separate in the
first place?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/5b1c8244/attachment.pgp>

From jedbrown at mcs.anl.gov  Tue Oct  1 18:42:05 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 18:42:05 -0500
Subject: [petsc-users] The product of two MPIDENSE matrices
In-Reply-To: <m2vc1gk5qs.wl%vishy@stat.purdue.edu>
References: <791899280.163311.1380661204449.JavaMail.root@mailhub028.itcs.purdue.edu>
	<87siwk3cmh.fsf@mcs.anl.gov> <m2vc1gk5qs.wl%vishy@stat.purdue.edu>
Message-ID: <874n9036eq.fsf@mcs.anl.gov>

S V N Vishwanathan <vishy at stat.purdue.edu> writes:

> Hi
>
>> > I am wondering whether I can multiply MATELEMENTAL and MATAIJ
>> > matrices. If I cannot multiply those, could you build the
>> > multiplication?
>> 
>> What are the dimensions of the matrices?
>> 
>> Jack, does Clique support multiplication of sparse matrices with
>> [MC,MR] dense matrices?
>
> The dense matrix is very flat and long (e.g. 10 x few 1000s) and the
> sparse matrix is also flat and very very long (e.g. few 1000s x few
> millions). I saw the code which is used to multiply a MPIAIJ matrix with
> a dense matrix. We need something very similar, but which uses elemental
> matrices. 

In practice, you'll probably get the (sparse) * (dense) version instead.
These aren't implemented yet for Elemental matrices, but you can
MatConvert to MPIDENSE and then do the sparse-dense product.  Why is the
original (few 1000s)x10 matrix in Elemental format?

PETSc does not currently support the [VC,*] distributions.  We can do
it, and maybe should, but it's not what the first users were asking for
and it's basically the same as MATDENSE except in its ability to
interact with [MC,MR].

> There seems to be some inconsistency right now in PETSc. I can multiple
> a MPIAIJ matrix with MPIDENSE, but I cannot multiply MPIDENSE with
> MPIDENSE since there is no PLAPACK support anymore. 

MPIDENSE was a bad distribution for squarish matrices.

> On other hand, I can multiply a MATELEMENTAL with MATELEMENTAL but I
> cannot multiply a MPIDENSE with MATELEMENTAL. Is the eventual plan to
> remove MPIDENSE completely and replace it with MATELEMENTAL?

Yes, at least partially.  Elemental is sort of a heavy dependency for
the large faction that only need Sparse x [VC,*].  For more general
distributions and some other algorithms (in the realm that PLAPACK was
used in the past), Elemental is the right way to go.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/af8930a2/attachment-0001.pgp>

From jedbrown at mcs.anl.gov  Tue Oct  1 18:47:41 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 18:47:41 -0500
Subject: [petsc-users] [dev@libelemental.org] Re: The product of two
	MPIDENSE matrices
In-Reply-To: <390172426.163780.1380668942300.JavaMail.root@mailhub028.itcs.purdue.edu>
References: <390172426.163780.1380668942300.JavaMail.root@mailhub028.itcs.purdue.edu>
Message-ID: <871u44365e.fsf@mcs.anl.gov>


Joon Hee Choi <choi240 at purdue.edu> writes:

> ----- Original Message -----
> From: "Jack Poulson" <poulson at stanford.edu>
> To: dev at libelemental.org, "Jed Brown" <jedbrown at mcs.anl.gov>
> Cc: "Joon Hee Choi" <choi240 at purdue.edu>, "Hong Zhang" <hzhang at mcs.anl.gov>, "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, October 1, 2013 6:15:23 PM
> Subject: Re: [dev at libelemental.org] Re: [petsc-users] The product of two MPIDENSE matrices

This quoting style is bulky and looks like a forward.  We have to
complain to vendors about things like this or they ruin everything.

> Yes, we have the similar multiplication as the form. 
> BTW, can we use the multiplication function with the MPIAIJ matrix we created in Petsc? Usually, the parallel matrix is row distributed matrix.

You can use MatTransposeMatMult with an MPIAIJ and a MPIDENSE.  (This is
the standard case mentioned earlier, and I don't understand what
Elemental is offering you in this situation.  Do you have other dense
matrices running around?)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/d0dee273/attachment.pgp>

From mirzadeh at gmail.com  Tue Oct  1 19:02:30 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 1 Oct 2013 17:02:30 -0700
Subject: [petsc-users] Scatter error
In-Reply-To: <87a9is37g8.fsf@mcs.anl.gov>
References: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>
	<87a9is37g8.fsf@mcs.anl.gov>
Message-ID: <CAKsPKwn3xyMzfQq0pc0S4nyGTgokvz7yknR=hfY7hKRtr6acJQ@mail.gmail.com>

> It would be half the messages.  Why are these vectors separate in the
> first place?

I had not worked with block vectors before so just convenience i
guess! When working with blocked vectors, how are items stored? Is it
like fxx[0], fyy[0], fxx[1], fyy[1], ... or is it like
fxx[0],fxx[1],...,fyy[0],fyy[1],...

From gryllida at fastmail.fm  Tue Oct  1 19:21:30 2013
From: gryllida at fastmail.fm (Gryllida)
Date: Wed, 02 Oct 2013 09:51:30 +0930
Subject: [petsc-users] Does PETSc have any broadcast function?
Message-ID: <1380673290.5300.28870661.28BF9FAC@webmail.messagingengine.com>

>On Wed, 1 Sep 2010 20:54:54 +0000, "Li, Zhisong (lizs)" <lizs at mail.uc.edu> wrote:
>>  Hi, Petsc Team,
>> 
>> I wonder if Petsc has any function like MPI_Bcast() which can
>> broadcast a value to all processes. If we directly add MPI functions
>> into the Petsc program, the MPI datatype may be incompatible with the
>> Petsc datatype. Does Petsc have any easy way to handle this?
>
>PETSc registers the types MPIU_SCALAR, MPIU_REAL, and MPIU_INT so you
>can use these to send PetscScalar, PetscReal, and PetscInt respectively
>using any MPI functions.
>
>Jed
>

What about MPI_DOUBLE?

From knepley at gmail.com  Tue Oct  1 19:26:17 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Oct 2013 19:26:17 -0500
Subject: [petsc-users] Scatter error
In-Reply-To: <CAKsPKwn3xyMzfQq0pc0S4nyGTgokvz7yknR=hfY7hKRtr6acJQ@mail.gmail.com>
References: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>
	<87a9is37g8.fsf@mcs.anl.gov>
	<CAKsPKwn3xyMzfQq0pc0S4nyGTgokvz7yknR=hfY7hKRtr6acJQ@mail.gmail.com>
Message-ID: <CAMYG4G=NHpYz5cJuKaUiXAOKAs0jzyRdEpqQb=TLGHV9oE8FfQ@mail.gmail.com>

On Tue, Oct 1, 2013 at 7:02 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> > It would be half the messages.  Why are these vectors separate in the
> > first place?
>
> I had not worked with block vectors before so just convenience i
> guess! When working with blocked vectors, how are items stored? Is it
> like fxx[0], fyy[0], fxx[1], fyy[1], ... or is it like
>

This one.

   Matt


> fxx[0],fxx[1],...,fyy[0],fyy[1],...
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/edb8aac6/attachment.html>

From mirzadeh at gmail.com  Tue Oct  1 19:35:24 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 1 Oct 2013 17:35:24 -0700
Subject: [petsc-users] Scatter error
In-Reply-To: <CAMYG4G=NHpYz5cJuKaUiXAOKAs0jzyRdEpqQb=TLGHV9oE8FfQ@mail.gmail.com>
References: <CAKsPKwnKQZX+uKMZzDNmV9Cj9-bbM_HOChJ4RHAeWXk3Z3ihLw@mail.gmail.com>
	<87a9is37g8.fsf@mcs.anl.gov>
	<CAKsPKwn3xyMzfQq0pc0S4nyGTgokvz7yknR=hfY7hKRtr6acJQ@mail.gmail.com>
	<CAMYG4G=NHpYz5cJuKaUiXAOKAs0jzyRdEpqQb=TLGHV9oE8FfQ@mail.gmail.com>
Message-ID: <CAKsPKw=4bR0N6dJ8EXRhLc+wS7n+uCi60FW1ssQgR9LDNt_25A@mail.gmail.com>

Thanks Matt.

On Tue, Oct 1, 2013 at 5:26 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Oct 1, 2013 at 7:02 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
>>
>> > It would be half the messages.  Why are these vectors separate in the
>> > first place?
>>
>> I had not worked with block vectors before so just convenience i
>> guess! When working with blocked vectors, how are items stored? Is it
>> like fxx[0], fyy[0], fxx[1], fyy[1], ... or is it like
>
>
> This one.
>
>    Matt
>
>>
>> fxx[0],fxx[1],...,fyy[0],fyy[1],...
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener

From knepley at gmail.com  Tue Oct  1 19:38:28 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 Oct 2013 19:38:28 -0500
Subject: [petsc-users] Does PETSc have any broadcast function?
In-Reply-To: <1380673290.5300.28870661.28BF9FAC@webmail.messagingengine.com>
References: <1380673290.5300.28870661.28BF9FAC@webmail.messagingengine.com>
Message-ID: <CAMYG4GkMyTGHWr7T271RZChV1owiVLQyC5g0J2=RWkiYMgwhbg@mail.gmail.com>

On Tue, Oct 1, 2013 at 7:21 PM, Gryllida <gryllida at fastmail.fm> wrote:

> >On Wed, 1 Sep 2010 20:54:54 +0000, "Li, Zhisong (lizs)" <lizs at
> mail.uc.edu> wrote:
> >>  Hi, Petsc Team,
> >>
> >> I wonder if Petsc has any function like MPI_Bcast() which can
> >> broadcast a value to all processes. If we directly add MPI functions
> >> into the Petsc program, the MPI datatype may be incompatible with the
> >> Petsc datatype. Does Petsc have any easy way to handle this?
> >
> >PETSc registers the types MPIU_SCALAR, MPIU_REAL, and MPIU_INT so you
> >can use these to send PetscScalar, PetscReal, and PetscInt respectively
> >using any MPI functions.
> >
> >Jed
> >
>
> What about MPI_DOUBLE?
>

That already exists.

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/388b582f/attachment.html>

From vishy at stat.purdue.edu  Tue Oct  1 20:03:20 2013
From: vishy at stat.purdue.edu (S V N Vishwanathan)
Date: Tue, 01 Oct 2013 21:03:20 -0400
Subject: [petsc-users] [dev@libelemental.org] Re: The product of two
	MPIDENSE matrices
In-Reply-To: <871u44365e.fsf@mcs.anl.gov>
References: <390172426.163780.1380668942300.JavaMail.root@mailhub028.itcs.purdue.edu>
	<871u44365e.fsf@mcs.anl.gov>
Message-ID: <m2siwkjxgn.wl%vishy@stat.purdue.edu>


> You can use MatTransposeMatMult with an MPIAIJ and a MPIDENSE.  (This
> is the standard case mentioned earlier, and I don't understand what
> Elemental is offering you in this situation.  Do you have other dense
> matrices running around?)

Perhaps I should give a bit of background on what we are trying to
achieve and the experts in the lists can give us some advice.

We are trying to find a k rank approximation to a n x m x p tensor
T. Here, n, m, p are of the order of millions but k is very small,
typically somewhere between 10 and 100. Also T is extremely sparse (nnz
is typically around 500 million or so).

Let A (size n x k), B(m x k) and C (p x k) be the factors. 

We are flattening the tensor along each of the three dimensions and
representing them as three matrices T1 (size n x mp), T2 (size m x np)
and T3 (size p x mn), all in MPIAIJ format.  

One of the intermediate computations that we perform is form the k x k
matrix A^T A. Another computation that we perform is to compute A^T T1.

If we set A to be MPIDENSE then PETSc cannot compute A^T A. So we
thought about using Elemental. But now the issue we are running into is
that we cannot form A^T T1 if A is of type MATELEMENTAL. 

Also somewhere in our code, we form a k x k matrix, collect it in one of
the processors and invert it and broadcast it back to all the other
processors. We need to use Cholesky or LU factorization for this
calculation. 

thanks

vishy



From wumengda at gmail.com  Tue Oct  1 20:05:11 2013
From: wumengda at gmail.com (Mengda Wu)
Date: Tue, 1 Oct 2013 21:05:11 -0400
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
	<alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
Message-ID: <CALmcJSCsvmyUdx_RjdSLqbXt_5D7oosaJk02M=6a_2MOOcsdHw@mail.gmail.com>

Hi Satish,

   Thanks a lot for your reply!

   Is there a way to use the mingw64 which is not shipped with cygwin? For
example, use windows native paths when testing the compiler.

Thanks,
Mengda


On Tue, Oct 1, 2013 at 5:55 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> Looks like this mingw install is not from cygwin - and does not work with
> cygwin paths.
>
> Suggest using mingw from cygwin - or install and use cygwin64 [which has
> non-mingw gcc-64]
> You might need the following patch
>
> https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207
>
> Satish
>
>
> On Tue, 1 Oct 2013, Mengda Wu wrote:
>
> > Hello all,
> >
> >    I am trying to compile 64 bit version of petsc-3.4.2 using MinGW-w64
> > (latest version 4.8.1 installed with
> >
> http://sourceforge.net/projects/mingwbuilds/files/mingw-builds-install/mingw-builds-install.exe/download
> )
> > on Windows 7 64 bit. I have cygwin, python (under cygwin) installed.
> >
> >   I run the "$ ./config/configure.py --with-cc='gcc' --with-fc='gfortran'
> > --with-cxx='g++'
> > --with-clanguage=cxx  --with-mpi=0 --with-debugging=0  --with-sowing=0
> > --with-c
> > 2html=0". I got some errors in checkCCompiler. The log file is as
> follows:
> > ============================================================
> > TEST checkCCompiler from
> >
> config.setCompilers(/cygdrive/c/Library/PETSc/download/petsc-3.4.2/config/BuildSystem/config/setCompilers.py:525)
> > TESTING: checkCCompiler from
> > config.setCompilers(config/BuildSystem/config/setCompilers.py:525)
> >   Locate a functional C compiler
> > Checking for program /home/glxiong/bin/gcc...not found
> > Checking for program /usr/local/bin/gcc...not found
> > Checking for program /usr/bin/gcc...not found
> > Checking for program
> > /cygdrive/c/Library/MinGW/x64-4.8.1/mingw64/bin/gcc...found
> >           Defined make macro "CC" to "gcc"
> >           Pushing language C
> >                     All intermediate test results are stored in
> > /tmp/petsc-novY5r/config.setCompilers
> > sh: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> > -I/tmp/petsc-novY5r/config.setCompilers
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c
> > Executing: gcc -c -o /tmp/petsc-novY5r/config.setCompilers/conftest.o
> > -I/tmp/petsc-novY5r/config.setCompilers
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c
> > sh:
> > Possible ERROR while running compiler: ret = 256
> > error message = {gcc.exe: error:
> > /tmp/petsc-novY5r/config.setCompilers/conftest.c: No such file or
> directory
> >
> > gcc.exe: fatal error: no input files
> >
> > compilation terminated.
> > ============================================================
> >
> > I think the problem is the MinGW-w64 compiler does not recognize the
> cygwin
> > path like /tmp. Is there a workaround? I tried to use a native Windows
> > python, but the configure fails to find the "make" command then. I guess
> > MinGW may be not supported by PetSc yet.
> >
> > However, I found someone has successfully compiled a MPI-version Petsc on
> > Windows 64 bit. Here is the link:
> > http://geuz.org/pipermail/gmsh/2012/007589.html.
> > How is that possible?
> >
> >
> > Thanks,
> > Mengda
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/4f830a10/attachment-0001.html>

From jedbrown at mcs.anl.gov  Tue Oct  1 20:17:24 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 20:17:24 -0500
Subject: [petsc-users] [dev@libelemental.org] Re: The product of two
	MPIDENSE matrices
In-Reply-To: <m2siwkjxgn.wl%vishy@stat.purdue.edu>
References: <390172426.163780.1380668942300.JavaMail.root@mailhub028.itcs.purdue.edu>
	<871u44365e.fsf@mcs.anl.gov> <m2siwkjxgn.wl%vishy@stat.purdue.edu>
Message-ID: <87ppro1nff.fsf@mcs.anl.gov>

S V N Vishwanathan <vishy at stat.purdue.edu> writes:

>> You can use MatTransposeMatMult with an MPIAIJ and a MPIDENSE.  (This
>> is the standard case mentioned earlier, and I don't understand what
>> Elemental is offering you in this situation.  Do you have other dense
>> matrices running around?)
>
> Perhaps I should give a bit of background on what we are trying to
> achieve and the experts in the lists can give us some advice.
>
> We are trying to find a k rank approximation to a n x m x p tensor
> T. Here, n, m, p are of the order of millions but k is very small,
> typically somewhere between 10 and 100. Also T is extremely sparse (nnz
> is typically around 500 million or so).
>
> Let A (size n x k), B(m x k) and C (p x k) be the factors. 
>
> We are flattening the tensor along each of the three dimensions and
> representing them as three matrices T1 (size n x mp), T2 (size m x np)
> and T3 (size p x mn), all in MPIAIJ format.  

Thanks for this description.

> One of the intermediate computations that we perform is form the k x k
> matrix A^T A. Another computation that we perform is to compute A^T T1.

I'm in favor of implementing the (very simple)
MatTransposeMatMult_MPIDense_MPIDense.  You would usually want the
result distributed redundantly, but PETSc doesn't have a dedicated
matrix format for redundant matrices.  We can add that (it would also be
useful in other places), but MatTransposeMatMult doesn't currently state
the desired output distribution and I want to make sure the bases are
covered if we do that.  In the mean time, you can use MatDenseGetArray,
dgemm, and MPI_Allreduce because that's what
MatTransposeMatMult_MPIDense_MPIDense would be doing when the result is
to be redundantly distributed.

> Also somewhere in our code, we form a k x k matrix, collect it in one of
> the processors and invert it and broadcast it back to all the other
> processors. We need to use Cholesky or LU factorization for this
> calculation. 

You might be better off forming the matrix redundantly (allreduce if
needed) and factoring it redundantly.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/32365c2c/attachment.pgp>

From mrosso at uci.edu  Tue Oct  1 21:02:28 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Tue, 01 Oct 2013 19:02:28 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
Message-ID: <524B7EB4.8040108@uci.edu>

Barry,

I repeated the previous runs since I noticed that it was not using the 
options

-mg_levels_ksp_max_it 3

They are faster then before now but still slower than my initial test 
and anyway the solution time increases considerably in time.
I attached the diagnostics (run1.txt and run2.txt, please see the files 
for the list of the options I used).
I also run a case by using your last proposed options (run3.txt): there 
is a divergence condition since 30 iterations seem not be enough to low 
the error below the need tolerance and thus after some time steps my 
solution blows up.
Please let me know what you think about it.
In the mean time I will try run my initial test with the option 
-mg_levels_ksp_max_it 3 instead of  -mg_levels_ksp_max_it 1.
As usual, thank you very much.

Michele

On 09/30/2013 07:17 PM, Barry Smith wrote:
>    I wasn't expecting this. Try
>
> -pc_mg_type full -ksp_type richardson -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3
> -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg
> -log_summary  -pc_mg_log  -ksp_monitor_true_residual  -options_left   -ksp_view -ksp_max_it 30
>
>
>
>
> On Sep 30, 2013, at 8:00 PM, Michele Rosso <mrosso at uci.edu> wrote:
>
>> Barry,
>>
>> sorry again for the very late answer. I tried all the variations you proposed: all of them converge very slow , except the last one (CG instead of fgmres) diverges.
>> I attached the diagnostics for the two options that convergence: each of the attached files starts with a list of the options I used for the run.
>> As you pointed out earlier, the residual I used to require was too small, therefore I increased atol to e-9.
>> After some tests, I noticed that any further increase of the absolute tolerance changes significantly the solution.
>> What would you suggest to try next?
>> Thank you very much,
>>
>> Michele
>>
>>
>>
>>
>>
>>
>> On 09/24/2013 05:08 PM, Barry Smith wrote:
>>>   Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate.
>>>
>>>   The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ????
>>>
>>>   Start with a really robust multigrid smoother use
>>>
>>> -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3   PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>> -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg
>>>
>>>   run with the -log_summary and -pc_mg_log
>>>
>>> Now back off a little on the smoother and use -mg_levels_pc_type sor instead  how does that change the convergence and time.
>>>
>>> Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson   how does that change the convergence and the time?
>>>
>>>   There are some additional variants we can try based on the results from above.
>>>
>>>   Barry
>>>
>>>
>>>
>>> On Sep 24, 2013, at 4:29 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>>
>>>> Barry,
>>>>
>>>> I re-rerun the test case with the option -pc_mg_log as you suggested.
>>>> I attached the new output ("final_new.txt').
>>>> Thanks for your help.
>>>>
>>>> Michele
>>>>
>>>> On 09/23/2013 09:35 AM, Barry Smith wrote:
>>>>>    Run with the additional option -pc_mg_log and send us the log file.
>>>>>
>>>>>    Barry
>>>>>
>>>>> Maybe we should make this the default somehow.
>>>>>
>>>>>
>>>>> On Sep 23, 2013, at 10:55 AM, Michele Rosso <mrosso at uci.edu> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG .  Such equation arises from a projection algorithm for a multiphase incompressible flow simulation.
>>>>>> I set up the solver  as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid).
>>>>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations.
>>>>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can.
>>>>>> Could you please help me out with this?
>>>>>>
>>>>>> I run the test case with the following options:
>>>>>>
>>>>>> -pc_type mg  -pc_mg_galerkin  -pc_mg_levels 5   -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
>>>>>> -mg_coarse_pc_type lu   -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>>>> -log_summary -ksp_view  -ksp_monitor_true_residual  -options_left
>>>>>>
>>>>>> Please find the diagnostic for the final solve in the attached file "final.txt'.
>>>>>> Thank you,
>>>>>>
>>>>>> Michele
>>>>>> <final.txt>
>>>> <final_new.txt>
>> <final1.txt><final2.txt>
>

-------------- next part --------------
OPTIONS:

-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg



  0 KSP unpreconditioned resid norm 1.036334906411e-06 true resid norm 1.036334906411e-06 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 7.265278566968e-07 true resid norm 7.265278566968e-07 ||r(i)||/||b|| 7.010550857663e-01
  2 KSP unpreconditioned resid norm 5.582868896378e-07 true resid norm 5.582868896378e-07 ||r(i)||/||b|| 5.387128101002e-01
  3 KSP unpreconditioned resid norm 3.634998226503e-07 true resid norm 3.634998226503e-07 ||r(i)||/||b|| 3.507551665022e-01
  4 KSP unpreconditioned resid norm 3.158574836963e-07 true resid norm 3.158574836963e-07 ||r(i)||/||b|| 3.047832141350e-01
  5 KSP unpreconditioned resid norm 2.777502658984e-07 true resid norm 2.777502658984e-07 ||r(i)||/||b|| 2.680120723331e-01
  6 KSP unpreconditioned resid norm 2.284215345140e-07 true resid norm 2.284215345140e-07 ||r(i)||/||b|| 2.204128540889e-01
  7 KSP unpreconditioned resid norm 1.969414140052e-07 true resid norm 1.969414140052e-07 ||r(i)||/||b|| 1.900364571210e-01
  8 KSP unpreconditioned resid norm 1.782653889853e-07 true resid norm 1.782653889853e-07 ||r(i)||/||b|| 1.720152316423e-01
  9 KSP unpreconditioned resid norm 1.536397911477e-07 true resid norm 1.536397911477e-07 ||r(i)||/||b|| 1.482530311362e-01
 10 KSP unpreconditioned resid norm 1.302441466920e-07 true resid norm 1.302441466920e-07 ||r(i)||/||b|| 1.256776606542e-01
 11 KSP unpreconditioned resid norm 1.142838199081e-07 true resid norm 1.142838199081e-07 ||r(i)||/||b|| 1.102769183988e-01
 12 KSP unpreconditioned resid norm 1.031156644411e-07 true resid norm 1.031156644411e-07 ||r(i)||/||b|| 9.950032928847e-02
 13 KSP unpreconditioned resid norm 9.027533291890e-08 true resid norm 9.027533291890e-08 ||r(i)||/||b|| 8.711019223651e-02
 14 KSP unpreconditioned resid norm 7.493550605983e-08 true resid norm 7.493550605983e-08 ||r(i)||/||b|| 7.230819457710e-02
 15 KSP unpreconditioned resid norm 6.742982059332e-08 true resid norm 6.742982059333e-08 ||r(i)||/||b|| 6.506566571884e-02
 16 KSP unpreconditioned resid norm 5.379988711018e-08 true resid norm 5.379988711019e-08 ||r(i)||/||b|| 5.191361091608e-02
 17 KSP unpreconditioned resid norm 4.872990226442e-08 true resid norm 4.872990226444e-08 ||r(i)||/||b|| 4.702138465372e-02
 18 KSP unpreconditioned resid norm 4.543576707229e-08 true resid norm 4.543576707230e-08 ||r(i)||/||b|| 4.384274503465e-02
 19 KSP unpreconditioned resid norm 4.243835036633e-08 true resid norm 4.243835036635e-08 ||r(i)||/||b|| 4.095042066401e-02
 20 KSP unpreconditioned resid norm 3.855396651833e-08 true resid norm 3.855396651834e-08 ||r(i)||/||b|| 3.720222707913e-02
 21 KSP unpreconditioned resid norm 3.540838965507e-08 true resid norm 3.540838965509e-08 ||r(i)||/||b|| 3.416693718994e-02
 22 KSP unpreconditioned resid norm 3.114021467696e-08 true resid norm 3.114021467699e-08 ||r(i)||/||b|| 3.004840856401e-02
 23 KSP unpreconditioned resid norm 2.687679086480e-08 true resid norm 2.687679086485e-08 ||r(i)||/||b|| 2.593446452356e-02
 24 KSP unpreconditioned resid norm 2.231320925522e-08 true resid norm 2.231320925524e-08 ||r(i)||/||b|| 2.153088650898e-02
 25 KSP unpreconditioned resid norm 1.849367046766e-08 true resid norm 1.849367046769e-08 ||r(i)||/||b|| 1.784526445387e-02
 26 KSP unpreconditioned resid norm 1.597347720732e-08 true resid norm 1.597347720735e-08 ||r(i)||/||b|| 1.541343161224e-02
 27 KSP unpreconditioned resid norm 1.351813033069e-08 true resid norm 1.351813033073e-08 ||r(i)||/||b|| 1.304417157726e-02
 28 KSP unpreconditioned resid norm 1.135895547453e-08 true resid norm 1.135895547456e-08 ||r(i)||/||b|| 1.096069948458e-02
 29 KSP unpreconditioned resid norm 9.644960881002e-09 true resid norm 9.644960881027e-09 ||r(i)||/||b|| 9.306799202997e-03
 30 KSP unpreconditioned resid norm 8.454149815651e-09 true resid norm 8.454149815651e-09 ||r(i)||/||b|| 8.157739127910e-03
 31 KSP unpreconditioned resid norm 7.380097753084e-09 true resid norm 7.380097753084e-09 ||r(i)||/||b|| 7.121344371812e-03
 32 KSP unpreconditioned resid norm 6.949063499474e-09 true resid norm 6.949063499474e-09 ||r(i)||/||b|| 6.705422596965e-03
 33 KSP unpreconditioned resid norm 6.732114039970e-09 true resid norm 6.732114039970e-09 ||r(i)||/||b|| 6.496079595816e-03
 34 KSP unpreconditioned resid norm 5.348043445752e-09 true resid norm 5.348043445752e-09 ||r(i)||/||b|| 5.160535858308e-03
 35 KSP unpreconditioned resid norm 4.753111075163e-09 true resid norm 4.753111075163e-09 ||r(i)||/||b|| 4.586462393342e-03
 36 KSP unpreconditioned resid norm 4.053751219961e-09 true resid norm 4.053751219961e-09 ||r(i)||/||b|| 3.911622772602e-03
 37 KSP unpreconditioned resid norm 3.648046750367e-09 true resid norm 3.648046750367e-09 ||r(i)||/||b|| 3.520142694990e-03
 38 KSP unpreconditioned resid norm 3.154002693751e-09 true resid norm 3.154002693751e-09 ||r(i)||/||b|| 3.043420301911e-03
 39 KSP unpreconditioned resid norm 2.796711364093e-09 true resid norm 2.796711364093e-09 ||r(i)||/||b|| 2.698655952618e-03
 40 KSP unpreconditioned resid norm 2.490859697022e-09 true resid norm 2.490859697022e-09 ||r(i)||/||b|| 2.403527741479e-03
 41 KSP unpreconditioned resid norm 2.214978925969e-09 true resid norm 2.214978925969e-09 ||r(i)||/||b|| 2.137319617689e-03
 42 KSP unpreconditioned resid norm 2.062308903139e-09 true resid norm 2.062308903139e-09 ||r(i)||/||b|| 1.990002353853e-03
 43 KSP unpreconditioned resid norm 1.893218419438e-09 true resid norm 1.893218419438e-09 ||r(i)||/||b|| 1.826840346422e-03
 44 KSP unpreconditioned resid norm 1.633410997565e-09 true resid norm 1.633410997565e-09 ||r(i)||/||b|| 1.576142024610e-03
 45 KSP unpreconditioned resid norm 1.497658367438e-09 true resid norm 1.497658367438e-09 ||r(i)||/||b|| 1.445149013290e-03
 46 KSP unpreconditioned resid norm 1.384720917112e-09 true resid norm 1.384720917112e-09 ||r(i)||/||b|| 1.336171259450e-03
 47 KSP unpreconditioned resid norm 1.149245204430e-09 true resid norm 1.149245204429e-09 ||r(i)||/||b|| 1.108951553517e-03
 48 KSP unpreconditioned resid norm 1.044541114051e-09 true resid norm 1.044541114051e-09 ||r(i)||/||b|| 1.007918490045e-03
 49 KSP unpreconditioned resid norm 8.702453784205e-10 true resid norm 8.702453784202e-10 ||r(i)||/||b|| 8.397337318626e-04
KSP Object: 128 MPI processes
  type: fgmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=0.001, absolute=1e-50, divergence=10000
  right preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object: 128 MPI processes
  type: mg
    MG: type is FULL, levels=5 cycles=v
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     128 MPI processes
      type: lu
        LU: out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 0, needed 0
          Factored matrix follows:
            Matrix Object:             128 MPI processes
              type: mpiaij
              rows=1024, cols=1024
              package used to perform factorization: superlu_dist
              total: nonzeros=0, allocated nonzeros=0
              total number of mallocs used during MatSetValues calls =0
                SuperLU_DIST run parameters:
                  Process grid nprow 16 x npcol 8 
                  Equilibrate matrix TRUE 
                  Matrix input mode 1 
                  Replace tiny pivots TRUE 
                  Use iterative refinement FALSE 
                  Processors in row 16 col partition 8 
                  Row permutation LargeDiag 
                  Column permutation METIS_AT_PLUS_A
                  Parallel symbolic factorization FALSE 
                  Repeated factorization SamePattern_SameRowPerm
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=1024, cols=1024
        total: nonzeros=27648, allocated nonzeros=27648
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_1_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_1_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=64, cols=64
                package used to perform factorization: petsc
                total: nonzeros=768, allocated nonzeros=768
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 16 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=64, cols=64
          total: nonzeros=768, allocated nonzeros=768
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 16 nodes, limit used is 5
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=8192, cols=8192
        total: nonzeros=221184, allocated nonzeros=221184
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 16 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_2_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_2_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=512, cols=512
                package used to perform factorization: petsc
                total: nonzeros=9600, allocated nonzeros=9600
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=512, cols=512
          total: nonzeros=9600, allocated nonzeros=9600
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=65536, cols=65536
        total: nonzeros=1769472, allocated nonzeros=1769472
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_3_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_3_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_3_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=4096, cols=4096
                package used to perform factorization: petsc
                total: nonzeros=92928, allocated nonzeros=92928
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=4096, cols=4096
          total: nonzeros=92928, allocated nonzeros=92928
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=524288, cols=524288
        total: nonzeros=14155776, allocated nonzeros=14155776
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_4_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_4_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_4_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=32768, cols=32768
                package used to perform factorization: petsc
                total: nonzeros=221184, allocated nonzeros=221184
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=32768, cols=32768
          total: nonzeros=221184, allocated nonzeros=221184
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=4194304, cols=4194304
        total: nonzeros=29360128, allocated nonzeros=29360128
        total number of mallocs used during MatSetValues calls =0
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   128 MPI processes
    type: mpiaij
    rows=4194304, cols=4194304
    total: nonzeros=29360128, allocated nonzeros=29360128
    total number of mallocs used during MatSetValues calls =0
 

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./hit on a interlagos-64idx-pgi-opt named nid12058 with 128 processors, by Unknown Tue Oct  1 12:47:42 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           5.564e+01      1.00031   5.563e+01
Objects:              1.344e+03      1.00000   1.344e+03
Flops:                7.519e+09      1.00000   7.519e+09  9.624e+11
Flops/sec:            1.352e+08      1.00031   1.352e+08  1.730e+10
MPI Messages:         2.584e+05      1.09854   2.354e+05  3.014e+07
MPI Message Lengths:  4.162e+08      1.00022   1.767e+03  5.326e+10
MPI Reductions:       4.880e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.4617e+01  26.3%  1.3758e+11  14.3%  4.615e+05   1.5%  2.117e+02       12.0%  3.681e+03   7.5% 
 1:        MG Apply: 4.1013e+01  73.7%  8.2479e+11  85.7%  2.968e+07  98.5%  1.556e+03       88.0%  4.512e+04  92.5% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecMDot              322 1.0 6.3548e-01 1.1 2.20e+08 1.0 0.0e+00 0.0e+00 3.2e+02  1  3  0  0  1   4 20  0  0  9 44353
VecNorm              786 1.0 2.7858e-01 2.9 5.15e+07 1.0 0.0e+00 0.0e+00 7.9e+02  0  1  0  0  2   1  5  0  0 21 23668
VecScale             372 1.0 1.7262e-02 1.1 1.22e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0 90388
VecCopy              414 1.0 5.8294e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               528 1.0 8.1342e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPY              368 1.0 6.1460e-02 1.3 2.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  2  0  0  0 50228
VecAYPX              368 1.0 4.9858e-02 1.5 1.21e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0 30958
VecWAXPY               4 1.0 1.1320e-03 2.1 1.31e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 14821
VecMAXPY             690 1.0 1.0875e+00 1.2 4.54e+08 1.0 0.0e+00 0.0e+00 0.0e+00  2  6  0  0  0   7 42  0  0  0 53393
VecScatterBegin      790 1.0 1.3550e-01 1.3 0.00e+00 0.0 3.8e+05 1.6e+04 0.0e+00  0  0  1 12  0   1  0 82 97  0     0
VecScatterEnd        790 1.0 3.5084e-01 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   2  0  0  0  0     0
MatMult              694 1.0 2.4980e+00 1.1 2.96e+08 1.0 3.6e+05 1.6e+04 0.0e+00  4  4  1 11  0  16 28 77 91  0 15149
MatMultTranspose       4 1.0 2.2562e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0 14338
MatLUFactorSym         1 1.0 5.0211e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 1.2263e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatAssemblyBegin      63 1.0 2.0496e-01 8.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02  0  0  0  0  0   1  0  0  0  3     0
MatAssemblyEnd        63 1.0 2.0343e-01 1.1 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01  0  0  0  0  0   1  0  3  0  2     0
MatGetRowIJ            1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.0981e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView              690 2.1 2.1315e-01 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.2e+02  0  0  0  0  1   1  0  0  0  9     0
MatPtAP                4 1.0 2.0783e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02  0  0  0  0  0   1  0  5  2  3  3144
MatPtAPSymbolic        4 1.0 1.4338e-01 1.0 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01  0  0  0  0  0   1  0  3  2  2     0
MatPtAPNumeric         4 1.0 7.0436e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01  0  0  0  0  0   0  0  2  0  1  9277
MatGetLocalMat         4 1.0 2.3359e-02 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          4 1.0 3.0560e-02 3.2 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00  0  0  0  0  0   0  0  2  1  0     0
MatGetSymTrans         8 1.0 1.0388e-02 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog       322 1.0 1.0764e+00 1.1 4.40e+08 1.0 0.0e+00 0.0e+00 3.2e+02  2  6  0  0  1   7 41  0  0  9 52372
KSPSetUp               6 1.0 3.7848e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01  0  0  0  0  0   0  0  0  0  2     0
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
PCSetUp                1 1.0 4.2101e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.3e+02  1  0  0  0  1   3  0  7  2  9  1629
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
MGSetup Level 0        1 1.0 1.2542e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   1  0  0  0  0     0
MGSetup Level 1        1 1.0 2.4819e-03 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 2        1 1.0 5.2500e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 3        1 1.0 5.1689e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 4        1 1.0 1.1669e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: MG Apply

VecMDot            19320 1.0 2.9361e+00 1.6 3.30e+08 1.0 0.0e+00 0.0e+00 1.9e+04  4  4  0  0 40   6  5  0  0 43 14401
VecNorm            25760 1.0 1.5193e+00 1.5 2.20e+08 1.0 0.0e+00 0.0e+00 2.6e+04  2  3  0  0 53   3  3  0  0 57 18556
VecScale           25760 1.0 1.7513e-01 1.1 1.10e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0 80490
VecCopy             8050 1.0 2.0040e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             46368 1.0 5.5128e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecAXPY            10948 1.0 1.9405e-01 1.2 1.07e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0 70388
VecAYPX             3220 1.0 5.9333e-02 1.1 1.38e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 29698
VecMAXPY           25760 1.0 5.2558e-01 1.1 4.96e+08 1.0 0.0e+00 0.0e+00 0.0e+00  1  7  0  0  0   1  8  0  0  0 120695
VecScatterBegin    36064 1.0 2.7229e+00 2.4 0.00e+00 0.0 3.0e+07 1.6e+03 0.0e+00  4  0 98 88  0   6  0100100  0     0
VecScatterEnd      36064 1.0 2.0120e+00 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   4  0  0  0  0     0
VecNormalize       25760 1.0 1.7211e+00 1.4 3.30e+08 1.0 0.0e+00 0.0e+00 2.6e+04  3  4  0  0 53   4  5  0  0 57 24571
MatMult            28336 1.0 1.9003e+01 1.1 2.75e+09 1.0 2.7e+07 1.7e+03 0.0e+00 33 37 89 85  0  45 43 90 96  0 18501
MatMultAdd          3220 1.0 7.4839e-01 1.2 9.29e+07 1.0 1.2e+06 5.3e+02 0.0e+00  1  1  4  1  0   2  1  4  1  0 15893
MatMultTranspose    4508 1.0 1.4513e+00 1.2 1.74e+08 1.0 1.7e+06 6.6e+02 0.0e+00  2  2  6  2  0   3  3  6  2  0 15372
MatSolve           27370 1.0 1.4336e+01 1.1 2.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 29  0  0  0  34 33  0  0  0 19206
MatLUFactorNum         4 1.0 1.0956e-02 1.2 1.86e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 21770
MatILUFactorSym        4 1.0 5.2656e-02 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            4 1.0 1.0967e-05 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         4 1.0 5.1585e-0230.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog     19320 1.0 3.2936e+00 1.5 6.61e+08 1.0 0.0e+00 0.0e+00 1.9e+04  5  9  0  0 40   7 10  0  0 43 25678
KSPSetUp               4 1.0 5.0068e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            8050 1.0 3.6157e+01 1.0 5.79e+09 1.0 2.3e+07 1.7e+03 4.5e+04 65 77 77 74 92  88 90 78 84100 20482
PCSetUp                4 1.0 7.6262e-02 2.3 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01  0  0  0  0  0   0  0  0  0  0  3127
PCSetUpOnBlocks     6440 1.0 8.1504e-02 2.1 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01  0  0  0  0  0   0  0  0  0  0  2926
PCApply            27370 1.0 1.5573e+01 1.1 2.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 27 29  0  0  0  37 33  0  0  0 17681
MGSmooth Level 0    1610 1.0 2.2521e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  4  0  0  0  0   5  0  0  0  0     0
MGSmooth Level 1    2576 1.0 1.4212e+00 1.0 5.43e+07 1.0 9.6e+06 1.9e+02 1.8e+04  3  1 32  3 37   3  1 32  4 40  4890
MGResid Level 1     1288 1.0 7.1253e-02 1.3 4.45e+06 1.0 1.3e+06 1.9e+02 0.0e+00  0  0  4  0  0   0  0  4  1  0  7996
MGInterp Level 1    3220 1.0 1.6670e-01 3.5 1.37e+06 1.0 1.2e+06 6.4e+01 0.0e+00  0  0  4  0  0   0  0  4  0  0  1052
MGSmooth Level 2    1932 1.0 2.2821e+00 1.0 3.82e+08 1.0 7.3e+06 6.4e+02 1.4e+04  4  5 24  9 28   5  6 24 10 30 21402
MGResid Level 2      966 1.0 1.4905e-01 1.5 2.67e+07 1.0 9.9e+05 6.4e+02 0.0e+00  0  0  3  1  0   0  0  3  1  0 22936
MGInterp Level 2    2576 1.0 1.8992e-01 2.3 8.74e+06 1.0 9.9e+05 2.1e+02 0.0e+00  0  0  3  0  0   0  0  3  0  0  5889
MGSmooth Level 3    1288 1.0 1.2010e+01 1.0 2.23e+09 1.0 4.9e+06 2.3e+03 9.0e+03 22 30 16 21 18  29 35 17 24 20 23726
MGResid Level 3      644 1.0 8.5489e-01 1.1 1.42e+08 1.0 6.6e+05 2.3e+03 0.0e+00  1  2  2  3  0   2  2  2  3  0 21327
MGInterp Level 3    1932 1.0 4.4246e-01 1.4 5.21e+07 1.0 7.4e+05 7.7e+02 0.0e+00  1  1  2  1  0   1  1  2  1  0 15071
MGSmooth Level 4     644 1.0 1.8477e+01 1.0 3.12e+09 1.0 1.3e+06 1.6e+04 4.5e+03 33 42  4 41  9  45 48  4 46 10 21640
MGResid Level 4      322 1.0 1.1910e+00 1.1 1.48e+08 1.0 1.6e+05 1.6e+04 0.0e+00  2  2  1  5  0   3  2  1  6  0 15876
MGInterp Level 4    1288 1.0 2.1723e+00 1.1 2.74e+08 1.0 4.9e+05 2.9e+03 0.0e+00  4  4  2  3  0   5  4  2  3  0 16165
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector   979            991    247462152     0
      Vector Scatter    19             19        22572     0
              Matrix    38             42     19508928     0
   Matrix Null Space     1              1          652     0
    Distributed Mesh     5              5       830792     0
     Bipartite Graph    10             10         8560     0
           Index Set    47             59       844496     0
   IS L to G Mapping     5              5       405756     0
       Krylov Solver    11             11       102272     0
     DMKSP interface     3              3         2088     0
      Preconditioner    11             11        11864     0
              Viewer   185            184       144256     0

--- Event Stage 1: MG Apply

              Vector    12              0            0     0
              Matrix     4              0            0     0
           Index Set    14              2         1792     0
========================================================================================================================
Average time to get PetscTime(): 2.14577e-07
Average time for MPI_Barrier(): 0.000448608
Average time for zero size MPI_Send(): 2.44565e-06
#PETSc Option Table entries:
-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure run at: Wed Aug 28 23:25:43 2013
Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok  --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
-----------------------------------------
Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 
Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2
Using PETSc arch: interlagos-64idx-pgi-opt
-----------------------------------------

Using C compiler: cc  -O3 -fastsse  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ftn  -O3 -fastsse   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include
-----------------------------------------

Using C linker: cc
Using Fortran linker: ftn
Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl 
-----------------------------------------

#PETSc Option Table entries:
-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
There are no unused options.
-------------- next part --------------
OPTIONS:
-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type sor
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg


  0 KSP unpreconditioned resid norm 5.609297891476e-07 true resid norm 5.609297891476e-07 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 2.581822560386e-07 true resid norm 2.581822560386e-07 ||r(i)||/||b|| 4.602755300819e-01
  2 KSP unpreconditioned resid norm 2.428085555532e-07 true resid norm 2.428085555532e-07 ||r(i)||/||b|| 4.328679992592e-01
  3 KSP unpreconditioned resid norm 2.335657385482e-07 true resid norm 2.335657385482e-07 ||r(i)||/||b|| 4.163903273940e-01
  4 KSP unpreconditioned resid norm 2.234855409323e-07 true resid norm 2.234855409323e-07 ||r(i)||/||b|| 3.984198116345e-01
  5 KSP unpreconditioned resid norm 2.078097503833e-07 true resid norm 2.078097503833e-07 ||r(i)||/||b|| 3.704737284484e-01
  6 KSP unpreconditioned resid norm 2.060299595561e-07 true resid norm 2.060299595561e-07 ||r(i)||/||b|| 3.673007986779e-01
  7 KSP unpreconditioned resid norm 1.304371586686e-07 true resid norm 1.304371586686e-07 ||r(i)||/||b|| 2.325374069842e-01
  8 KSP unpreconditioned resid norm 1.282069723612e-07 true resid norm 1.282069723612e-07 ||r(i)||/||b|| 2.285615327294e-01
  9 KSP unpreconditioned resid norm 8.994010091283e-08 true resid norm 8.994010091283e-08 ||r(i)||/||b|| 1.603411026708e-01
 10 KSP unpreconditioned resid norm 7.624200700050e-08 true resid norm 7.624200700050e-08 ||r(i)||/||b|| 1.359207666905e-01
 11 KSP unpreconditioned resid norm 7.614198614178e-08 true resid norm 7.614198614178e-08 ||r(i)||/||b|| 1.357424540734e-01
 12 KSP unpreconditioned resid norm 6.892442767094e-08 true resid norm 6.892442767094e-08 ||r(i)||/||b|| 1.228753205917e-01
 13 KSP unpreconditioned resid norm 6.891270426886e-08 true resid norm 6.891270426886e-08 ||r(i)||/||b|| 1.228544206461e-01
 14 KSP unpreconditioned resid norm 5.299112064401e-08 true resid norm 5.299112064401e-08 ||r(i)||/||b|| 9.447014879446e-02
 15 KSP unpreconditioned resid norm 4.214889484631e-08 true resid norm 4.214889484631e-08 ||r(i)||/||b|| 7.514112400835e-02
 16 KSP unpreconditioned resid norm 2.789939104957e-08 true resid norm 2.789939104957e-08 ||r(i)||/||b|| 4.973775967928e-02
 17 KSP unpreconditioned resid norm 2.786722600854e-08 true resid norm 2.786722600854e-08 ||r(i)||/||b|| 4.968041731371e-02
 18 KSP unpreconditioned resid norm 2.457366893493e-08 true resid norm 2.457366893493e-08 ||r(i)||/||b|| 4.380881424085e-02
 19 KSP unpreconditioned resid norm 2.430122634853e-08 true resid norm 2.430122634853e-08 ||r(i)||/||b|| 4.332311604534e-02
 20 KSP unpreconditioned resid norm 1.694910033682e-08 true resid norm 1.694910033683e-08 ||r(i)||/||b|| 3.021608169997e-02
 21 KSP unpreconditioned resid norm 1.383837294163e-08 true resid norm 1.383837294164e-08 ||r(i)||/||b|| 2.467041902458e-02
 22 KSP unpreconditioned resid norm 1.156264774412e-08 true resid norm 1.156264774414e-08 ||r(i)||/||b|| 2.061336011715e-02
 23 KSP unpreconditioned resid norm 7.091619620968e-09 true resid norm 7.091619620978e-09 ||r(i)||/||b|| 1.264261545416e-02
 24 KSP unpreconditioned resid norm 7.065272914870e-09 true resid norm 7.065272914878e-09 ||r(i)||/||b|| 1.259564575027e-02
 25 KSP unpreconditioned resid norm 6.618876503623e-09 true resid norm 6.618876503634e-09 ||r(i)||/||b|| 1.179983062353e-02
 26 KSP unpreconditioned resid norm 6.462504586080e-09 true resid norm 6.462504586091e-09 ||r(i)||/||b|| 1.152105791335e-02
 27 KSP unpreconditioned resid norm 6.440405749487e-09 true resid norm 6.440405749498e-09 ||r(i)||/||b|| 1.148166111713e-02
 28 KSP unpreconditioned resid norm 5.744119795026e-09 true resid norm 5.744119795044e-09 ||r(i)||/||b|| 1.024035432986e-02
 29 KSP unpreconditioned resid norm 2.587935833329e-09 true resid norm 2.587935833348e-09 ||r(i)||/||b|| 4.613653764548e-03
 30 KSP unpreconditioned resid norm 7.019604467372e-10 true resid norm 7.019604467372e-10 ||r(i)||/||b|| 1.251423012859e-03
 31 KSP unpreconditioned resid norm 7.001655404576e-10 true resid norm 7.001655404576e-10 ||r(i)||/||b|| 1.248223135950e-03
 32 KSP unpreconditioned resid norm 6.876834537207e-10 true resid norm 6.876834537207e-10 ||r(i)||/||b|| 1.225970642004e-03
 33 KSP unpreconditioned resid norm 6.378517201634e-10 true resid norm 6.378517201634e-10 ||r(i)||/||b|| 1.137132904160e-03
 34 KSP unpreconditioned resid norm 5.450153316415e-10 true resid norm 5.450153316415e-10 ||r(i)||/||b|| 9.716284322674e-04
KSP Object: 128 MPI processes
  type: fgmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=0.001, absolute=1e-50, divergence=10000
  right preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object: 128 MPI processes
  type: mg
    MG: type is FULL, levels=5 cycles=v
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     128 MPI processes
      type: lu
        LU: out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 0, needed 0
          Factored matrix follows:
            Matrix Object:             128 MPI processes
              type: mpiaij
              rows=1024, cols=1024
              package used to perform factorization: superlu_dist
              total: nonzeros=0, allocated nonzeros=0
              total number of mallocs used during MatSetValues calls =0
                SuperLU_DIST run parameters:
                  Process grid nprow 16 x npcol 8 
                  Equilibrate matrix TRUE 
                  Matrix input mode 1 
                  Replace tiny pivots TRUE 
                  Use iterative refinement FALSE 
                  Processors in row 16 col partition 8 
                  Row permutation LargeDiag 
                  Column permutation METIS_AT_PLUS_A
                  Parallel symbolic factorization FALSE 
                  Repeated factorization SamePattern_SameRowPerm
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=1024, cols=1024
        total: nonzeros=27648, allocated nonzeros=27648
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=8192, cols=8192
        total: nonzeros=221184, allocated nonzeros=221184
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 16 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=65536, cols=65536
        total: nonzeros=1769472, allocated nonzeros=1769472
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_3_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=524288, cols=524288
        total: nonzeros=14155776, allocated nonzeros=14155776
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_4_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=4194304, cols=4194304
        total: nonzeros=29360128, allocated nonzeros=29360128
        total number of mallocs used during MatSetValues calls =0
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   128 MPI processes
    type: mpiaij
    rows=4194304, cols=4194304
    total: nonzeros=29360128, allocated nonzeros=29360128
    total number of mallocs used during MatSetValues calls =0
 

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./hit on a interlagos-64idx-pgi-opt named nid12058 with 128 processors, by Unknown Tue Oct  1 13:05:12 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.265e+02      1.00006   2.265e+02
Objects:              3.366e+03      1.00000   3.366e+03
Flops:                2.626e+10      1.00000   2.626e+10  3.361e+12
Flops/sec:            1.159e+08      1.00006   1.159e+08  1.484e+10
MPI Messages:         8.260e+05      1.00000   8.260e+05  1.057e+08
MPI Message Lengths:  1.464e+09      1.00000   1.773e+03  1.874e+11
MPI Reductions:       1.710e+05      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 7.1195e+01  31.4%  4.1158e+11  12.2%  1.404e+06   1.3%  2.136e+02       12.1%  1.244e+04   7.3% 
 1:        MG Apply: 1.5530e+02  68.6%  2.9491e+12  87.8%  1.043e+08  98.7%  1.559e+03       87.9%  1.585e+05  92.7% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecMDot             1132 1.0 1.6774e+00 1.2 5.77e+08 1.0 0.0e+00 0.0e+00 1.1e+03  1  2  0  0  1   2 18  0  0  9 44049
VecNorm             2873 1.0 9.3535e-01 2.5 1.88e+08 1.0 0.0e+00 0.0e+00 2.9e+03  0  1  0  0  2   1  6  0  0 23 25766
VecScale            1339 1.0 6.0955e-02 1.0 4.39e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0 92136
VecCopy             1534 1.0 2.0990e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              1958 1.0 3.0763e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY             1333 1.0 2.2397e-01 1.3 8.74e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  3  0  0  0 49926
VecAYPX             1333 1.0 1.7871e-01 1.4 4.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0 31285
VecWAXPY               6 1.0 1.6773e-03 1.9 1.97e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 15004
VecMAXPY            2465 1.0 2.9482e+00 1.2 1.22e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  5  0  0  0   4 38  0  0  0 52831
VecScatterBegin     2877 1.0 5.2887e-01 1.3 0.00e+00 0.0 1.4e+06 1.6e+04 0.0e+00  0  0  1 12  0   1  0 98 99  0     0
VecScatterEnd       2877 1.0 1.1966e+00 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMult             2471 1.0 8.7551e+00 1.1 1.05e+09 1.0 1.3e+06 1.6e+04 0.0e+00  4  4  1 11  0  12 33 90 92  0 15389
MatMultTranspose       4 1.0 2.2500e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0 14377
MatLUFactorSym         1 1.0 5.0402e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 1.2829e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin     218 1.0 8.1404e-0118.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.2e+02  0  0  0  0  0   1  0  0  0  3     0
MatAssemblyEnd       218 1.0 7.6480e-01 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01  0  0  0  0  0   1  0  1  0  1     0
MatGetRowIJ            1 1.0 3.0994e-06 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.2173e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView             1407 1.0 4.3674e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 1.4e+03  0  0  0  0  1   1  0  0  0 11     0
MatPtAP                4 1.0 2.1245e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02  0  0  0  0  0   0  0  2  1  1  3076
MatPtAPSymbolic        4 1.0 1.4296e-01 1.0 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01  0  0  0  0  0   0  0  1  1  0     0
MatPtAPNumeric         4 1.0 7.0840e-02 1.0 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01  0  0  0  0  0   0  0  1  0  0  9224
MatGetLocalMat         4 1.0 2.2167e-02 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          4 1.0 2.9867e-02 3.1 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00  0  0  0  0  0   0  0  1  0  0     0
MatGetSymTrans         8 1.0 9.5510e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog      1132 1.0 2.8047e+00 1.1 1.15e+09 1.0 0.0e+00 0.0e+00 1.1e+03  1  4  0  0  1   4 36  0  0  9 52688
KSPSetUp               6 1.0 3.5489e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+01  0  0  0  0  0   0  0  0  0  0     0
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
PCSetUp                1 1.0 4.2664e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.2e+02  0  0  0  0  0   1  0  2  1  3  1607
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
MGSetup Level 0        1 1.0 1.3110e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 1        1 1.0 1.7891e-03 7.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 2        1 1.0 2.5201e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 3        1 1.0 3.9697e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 4        1 1.0 1.0642e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: MG Apply

VecMDot            67920 1.0 1.0526e+01 1.5 1.16e+09 1.0 0.0e+00 0.0e+00 6.8e+04  4  4  0  0 40   6  5  0  0 43 14122
VecNorm            90560 1.0 5.5366e+00 1.4 7.74e+08 1.0 0.0e+00 0.0e+00 9.1e+04  2  3  0  0 53   3  3  0  0 57 17901
VecScale           90560 1.0 6.1576e-01 1.1 3.87e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0 80480
VecCopy            28300 1.0 7.0869e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             72448 1.0 5.1643e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            38488 1.0 7.2360e-01 1.2 3.75e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  2  0  0  0 66359
VecAYPX            11320 1.0 2.0807e-01 1.1 4.84e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 29771
VecMAXPY           90560 1.0 1.7752e+00 1.1 1.74e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1  7  0  0  0   1  8  0  0  0 125622
VecScatterBegin   126784 1.0 9.4371e+00 2.3 0.00e+00 0.0 1.0e+08 1.6e+03 0.0e+00  4  0 99 88  0   5  0100100  0     0
VecScatterEnd     126784 1.0 6.8194e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   4  0  0  0  0     0
VecNormalize       90560 1.0 6.2612e+00 1.4 1.16e+09 1.0 0.0e+00 0.0e+00 9.1e+04  2  4  0  0 53   4  5  0  0 57 23745
MatMult            99616 1.0 5.7795e+01 1.1 9.66e+09 1.0 9.4e+07 1.7e+03 0.0e+00 25 37 89 85  0  36 42 90 96  0 21385
MatMultAdd         11320 1.0 2.6260e+00 1.2 3.27e+08 1.0 4.3e+06 5.3e+02 0.0e+00  1  1  4  1  0   2  1  4  1  0 15923
MatMultTranspose   15848 1.0 4.8380e+00 1.1 6.13e+08 1.0 6.1e+06 6.6e+02 0.0e+00  2  2  6  2  0   3  3  6  2  0 16212
MatSolve            5660 1.0 7.7166e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   5  0  0  0  0     0
MatSOR             90560 1.0 6.6838e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 29 30  0  0  0  42 35  0  0  0 15237
KSPGMRESOrthog     67920 1.0 1.1756e+01 1.4 2.32e+09 1.0 0.0e+00 0.0e+00 6.8e+04  4  9  0  0 40   7 10  0  0 43 25290
KSPSolve           28300 1.0 1.3913e+02 1.0 2.07e+10 1.0 8.1e+07 1.7e+03 1.6e+05 61 79 77 74 93  89 90 78 84100 19070
PCApply            96220 1.0 7.4543e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 32 30  0  0  0  47 35  0  0  0 13662
MGSmooth Level 0    5660 1.0 7.8165e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   5  0  0  0  0     0
MGSmooth Level 1    9056 1.0 3.9531e+00 1.1 1.93e+08 1.0 3.4e+07 1.9e+02 6.3e+04  2  1 32  3 37   2  1 32  4 40  6255
MGResid Level 1     4528 1.0 2.5980e-01 1.4 1.56e+07 1.0 4.6e+06 1.9e+02 0.0e+00  0  0  4  0  0   0  0  4  1  0  7710
MGInterp Level 1   11320 1.0 8.6609e-01 5.1 4.82e+06 1.0 4.3e+06 6.4e+01 0.0e+00  0  0  4  0  0   0  0  4  0  0   712
MGSmooth Level 2    6792 1.0 7.4435e+00 1.0 1.36e+09 1.0 2.6e+07 6.4e+02 4.8e+04  3  5 24  9 28   5  6 24 10 30 23301
MGResid Level 2     3396 1.0 5.4746e-01 1.4 9.39e+07 1.0 3.5e+06 6.4e+02 0.0e+00  0  0  3  1  0   0  0  3  1  0 21953
MGInterp Level 2    9056 1.0 6.0639e-01 2.0 3.07e+07 1.0 3.5e+06 2.1e+02 0.0e+00  0  0  3  0  0   0  0  3  0  0  6484
MGSmooth Level 3    4528 1.0 3.5954e+01 1.0 7.90e+09 1.0 1.7e+07 2.3e+03 3.2e+04 16 30 16 21 19  23 34 17 24 20 28111
MGResid Level 3     2264 1.0 2.0338e+00 1.2 5.01e+08 1.0 2.3e+06 2.3e+03 0.0e+00  1  2  2  3  0   1  2  2  3  0 31516
MGInterp Level 3    6792 1.0 1.5462e+00 1.4 1.83e+08 1.0 2.6e+06 7.7e+02 0.0e+00  1  1  2  1  0   1  1  2  1  0 15161
MGSmooth Level 4    2264 1.0 8.5215e+01 1.0 1.13e+10 1.0 4.6e+06 1.6e+04 1.6e+04 38 43  4 41  9  55 49  4 46 10 16948
MGResid Level 4     1132 1.0 4.1353e+00 1.1 5.19e+08 1.0 5.8e+05 1.6e+04 0.0e+00  2  2  1  5  0   3  2  1  6  0 16074
MGInterp Level 4    4528 1.0 7.5078e+00 1.1 9.64e+08 1.0 1.7e+06 2.9e+03 0.0e+00  3  4  2  3  0   5  4  2  3  0 16442
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector  3211           3223    861601128     0
      Vector Scatter    19             19        22572     0
              Matrix    38             38     14004608     0
   Matrix Null Space     1              1          652     0
    Distributed Mesh     5              5       830792     0
     Bipartite Graph    10             10         8560     0
           Index Set    47             47       534480     0
   IS L to G Mapping     5              5       405756     0
       Krylov Solver     7              7        97216     0
     DMKSP interface     3              3         2088     0
      Preconditioner     7              7         7352     0
              Viewer     1              0            0     0

--- Event Stage 1: MG Apply

              Vector    12              0            0     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 1.62125e-05
Average time for zero size MPI_Send(): 2.36742e-06
#PETSc Option Table entries:
-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type sor
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure run at: Wed Aug 28 23:25:43 2013
Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok  --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
-----------------------------------------
Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 
Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2
Using PETSc arch: interlagos-64idx-pgi-opt
-----------------------------------------

Using C compiler: cc  -O3 -fastsse  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ftn  -O3 -fastsse   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include
-----------------------------------------

Using C linker: cc
Using Fortran linker: ftn
Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl 
-----------------------------------------

#PETSc Option Table entries:
-ksp_monitor_true_residual
-ksp_type fgmres
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type sor
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
There are no unused options.
-------------- next part --------------
OPTIONS:
-ksp_atol 1e-9
-ksp_max_it 30
-ksp_monitor_true_residual
-ksp_type richardson
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg


  0 KSP unpreconditioned resid norm 6.954195782521e-06 true resid norm 6.954195782521e-06 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 4.019686111644e-06 true resid norm 4.019686111644e-06 ||r(i)||/||b|| 5.780231442070e-01
  2 KSP unpreconditioned resid norm 3.085633839543e-06 true resid norm 3.085633839543e-06 ||r(i)||/||b|| 4.437082210568e-01
  3 KSP unpreconditioned resid norm 3.108638870884e-06 true resid norm 3.108638870884e-06 ||r(i)||/||b|| 4.470163003891e-01
  4 KSP unpreconditioned resid norm 2.907823260441e-06 true resid norm 2.907823260441e-06 ||r(i)||/||b|| 4.181394012158e-01
  5 KSP unpreconditioned resid norm 3.095105825237e-06 true resid norm 3.095105825237e-06 ||r(i)||/||b|| 4.450702744114e-01
  6 KSP unpreconditioned resid norm 2.816885443509e-06 true resid norm 2.816885443509e-06 ||r(i)||/||b|| 4.050627177608e-01
  7 KSP unpreconditioned resid norm 3.330756254322e-06 true resid norm 3.330756254322e-06 ||r(i)||/||b|| 4.789563536151e-01
  8 KSP unpreconditioned resid norm 2.734103597927e-06 true resid norm 2.734103597927e-06 ||r(i)||/||b|| 3.931588473249e-01
  9 KSP unpreconditioned resid norm 3.029844525689e-06 true resid norm 3.029844525689e-06 ||r(i)||/||b|| 4.356858248519e-01
 10 KSP unpreconditioned resid norm 2.626258637892e-06 true resid norm 2.626258637892e-06 ||r(i)||/||b|| 3.776509491569e-01
 11 KSP unpreconditioned resid norm 2.620796722232e-06 true resid norm 2.620796722232e-06 ||r(i)||/||b|| 3.768655361731e-01
 12 KSP unpreconditioned resid norm 2.599366584696e-06 true resid norm 2.599366584696e-06 ||r(i)||/||b|| 3.737839235457e-01
 13 KSP unpreconditioned resid norm 2.815136272808e-06 true resid norm 2.815136272808e-06 ||r(i)||/||b|| 4.048111903728e-01
 14 KSP unpreconditioned resid norm 2.592704976330e-06 true resid norm 2.592704976330e-06 ||r(i)||/||b|| 3.728259970544e-01
 15 KSP unpreconditioned resid norm 2.647297548295e-06 true resid norm 2.647297548295e-06 ||r(i)||/||b|| 3.806763040737e-01
 16 KSP unpreconditioned resid norm 2.577657729007e-06 true resid norm 2.577657729007e-06 ||r(i)||/||b|| 3.706622317833e-01
 17 KSP unpreconditioned resid norm 2.637186195877e-06 true resid norm 2.637186195877e-06 ||r(i)||/||b|| 3.792223110120e-01
 18 KSP unpreconditioned resid norm 2.569979492081e-06 true resid norm 2.569979492081e-06 ||r(i)||/||b|| 3.695581160572e-01
 19 KSP unpreconditioned resid norm 2.639092183189e-06 true resid norm 2.639092183189e-06 ||r(i)||/||b|| 3.794963883275e-01
 20 KSP unpreconditioned resid norm 2.557359938672e-06 true resid norm 2.557359938672e-06 ||r(i)||/||b|| 3.677434485091e-01
 21 KSP unpreconditioned resid norm 2.619919367497e-06 true resid norm 2.619919367497e-06 ||r(i)||/||b|| 3.767393742469e-01
 22 KSP unpreconditioned resid norm 2.540615865281e-06 true resid norm 2.540615865281e-06 ||r(i)||/||b|| 3.653356829077e-01
 23 KSP unpreconditioned resid norm 2.578329382313e-06 true resid norm 2.578329382313e-06 ||r(i)||/||b|| 3.707588142389e-01
 24 KSP unpreconditioned resid norm 2.525920830833e-06 true resid norm 2.525920830833e-06 ||r(i)||/||b|| 3.632225651716e-01
 25 KSP unpreconditioned resid norm 2.658560319798e-06 true resid norm 2.658560319798e-06 ||r(i)||/||b|| 3.822958689890e-01
 26 KSP unpreconditioned resid norm 2.522426607571e-06 true resid norm 2.522426607571e-06 ||r(i)||/||b|| 3.627201025762e-01
 27 KSP unpreconditioned resid norm 2.616030191476e-06 true resid norm 2.616030191476e-06 ||r(i)||/||b|| 3.761801182031e-01
 28 KSP unpreconditioned resid norm 2.507602013260e-06 true resid norm 2.507602013260e-06 ||r(i)||/||b|| 3.605883543806e-01
 29 KSP unpreconditioned resid norm 2.624604598576e-06 true resid norm 2.624604598576e-06 ||r(i)||/||b|| 3.774131014793e-01
 30 KSP unpreconditioned resid norm 2.502026180934e-06 true resid norm 2.502026180934e-06 ||r(i)||/||b|| 3.597865603990e-01
KSP Object: 128 MPI processes
  type: richardson
    Richardson: damping factor=1
  maximum iterations=30, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-09, divergence=10000
  left preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object: 128 MPI processes
  type: mg
    MG: type is FULL, levels=5 cycles=v
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     128 MPI processes
      type: lu
        LU: out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 0, needed 0
          Factored matrix follows:
            Matrix Object:             128 MPI processes
              type: mpiaij
              rows=1024, cols=1024
              package used to perform factorization: superlu_dist
              total: nonzeros=0, allocated nonzeros=0
              total number of mallocs used during MatSetValues calls =0
                SuperLU_DIST run parameters:
                  Process grid nprow 16 x npcol 8 
                  Equilibrate matrix TRUE 
                  Matrix input mode 1 
                  Replace tiny pivots TRUE 
                  Use iterative refinement FALSE 
                  Processors in row 16 col partition 8 
                  Row permutation LargeDiag 
                  Column permutation METIS_AT_PLUS_A
                  Parallel symbolic factorization FALSE 
                  Repeated factorization SamePattern_SameRowPerm
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=1024, cols=1024
        total: nonzeros=27648, allocated nonzeros=27648
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_1_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_1_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=64, cols=64
                package used to perform factorization: petsc
                total: nonzeros=768, allocated nonzeros=768
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 16 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=64, cols=64
          total: nonzeros=768, allocated nonzeros=768
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 16 nodes, limit used is 5
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=8192, cols=8192
        total: nonzeros=221184, allocated nonzeros=221184
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 16 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_2_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_2_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=512, cols=512
                package used to perform factorization: petsc
                total: nonzeros=9600, allocated nonzeros=9600
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=512, cols=512
          total: nonzeros=9600, allocated nonzeros=9600
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=65536, cols=65536
        total: nonzeros=1769472, allocated nonzeros=1769472
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_3_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_3_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_3_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=4096, cols=4096
                package used to perform factorization: petsc
                total: nonzeros=92928, allocated nonzeros=92928
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=4096, cols=4096
          total: nonzeros=92928, allocated nonzeros=92928
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=524288, cols=524288
        total: nonzeros=14155776, allocated nonzeros=14155776
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_4_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_4_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_4_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=32768, cols=32768
                package used to perform factorization: petsc
                total: nonzeros=221184, allocated nonzeros=221184
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=32768, cols=32768
          total: nonzeros=221184, allocated nonzeros=221184
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=4194304, cols=4194304
        total: nonzeros=29360128, allocated nonzeros=29360128
        total number of mallocs used during MatSetValues calls =0
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   128 MPI processes
    type: mpiaij
    rows=4194304, cols=4194304
    total: nonzeros=29360128, allocated nonzeros=29360128
    total number of mallocs used during MatSetValues calls =0

  0 KSP unpreconditioned resid norm 2.917180555663e-04 true resid norm 2.917180555663e-04 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 1.049878179956e-02 true resid norm 1.049878179956e-02 ||r(i)||/||b|| 3.598948230742e+01
  2 KSP unpreconditioned resid norm 4.603139618725e-01 true resid norm 4.603139618725e-01 ||r(i)||/||b|| 1.577941279565e+03
  3 KSP unpreconditioned resid norm 2.274779569665e+01 true resid norm 2.274779569665e+01 ||r(i)||/||b|| 7.797870328075e+04
KSP Object: 128 MPI processes
  type: richardson
    Richardson: damping factor=1
  maximum iterations=30, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-09, divergence=10000
  left preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object: 128 MPI processes
  type: mg
    MG: type is FULL, levels=5 cycles=v
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     128 MPI processes
      type: lu
        LU: out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 0, needed 0
          Factored matrix follows:
            Matrix Object:             128 MPI processes
              type: mpiaij
              rows=1024, cols=1024
              package used to perform factorization: superlu_dist
              total: nonzeros=0, allocated nonzeros=0
              total number of mallocs used during MatSetValues calls =0
                SuperLU_DIST run parameters:
                  Process grid nprow 16 x npcol 8 
                  Equilibrate matrix TRUE 
                  Matrix input mode 1 
                  Replace tiny pivots TRUE 
                  Use iterative refinement FALSE 
                  Processors in row 16 col partition 8 
                  Row permutation LargeDiag 
                  Column permutation METIS_AT_PLUS_A
                  Parallel symbolic factorization FALSE 
                  Repeated factorization SamePattern_SameRowPerm
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=1024, cols=1024
        total: nonzeros=27648, allocated nonzeros=27648
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_1_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_1_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=64, cols=64
                package used to perform factorization: petsc
                total: nonzeros=768, allocated nonzeros=768
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 16 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=64, cols=64
          total: nonzeros=768, allocated nonzeros=768
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 16 nodes, limit used is 5
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=8192, cols=8192
        total: nonzeros=221184, allocated nonzeros=221184
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 16 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_2_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_2_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=512, cols=512
                package used to perform factorization: petsc
                total: nonzeros=9600, allocated nonzeros=9600
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=512, cols=512
          total: nonzeros=9600, allocated nonzeros=9600
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=65536, cols=65536
        total: nonzeros=1769472, allocated nonzeros=1769472
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_3_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_3_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_3_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=4096, cols=4096
                package used to perform factorization: petsc
                total: nonzeros=92928, allocated nonzeros=92928
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=4096, cols=4096
          total: nonzeros=92928, allocated nonzeros=92928
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=524288, cols=524288
        total: nonzeros=14155776, allocated nonzeros=14155776
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     128 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_4_)     128 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 128
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_4_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_4_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=32768, cols=32768
                package used to perform factorization: petsc
                total: nonzeros=221184, allocated nonzeros=221184
                total number of mallocs used during MatSetValues calls =0
                  not using I-node routines
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=32768, cols=32768
          total: nonzeros=221184, allocated nonzeros=221184
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=4194304, cols=4194304
        total: nonzeros=29360128, allocated nonzeros=29360128
        total number of mallocs used during MatSetValues calls =0
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   128 MPI processes
    type: mpiaij
    rows=4194304, cols=4194304
    total: nonzeros=29360128, allocated nonzeros=29360128
    total number of mallocs used during MatSetValues calls =0
 

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./hit on a interlagos-64idx-pgi-opt named nid25319 with 128 processors, by Unknown Tue Oct  1 20:23:44 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           4.942e+01      1.00014   4.942e+01
Objects:              1.206e+03      1.00000   1.206e+03
Flops:                6.078e+09      1.00000   6.078e+09  7.779e+11
Flops/sec:            1.230e+08      1.00014   1.230e+08  1.574e+10
MPI Messages:         2.322e+05      1.11091   2.092e+05  2.678e+07
MPI Message Lengths:  3.703e+08      1.00025   1.769e+03  4.739e+10
MPI Reductions:       4.337e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.3192e+01  26.7%  4.5330e+10   5.8%  4.225e+05   1.6%  2.144e+02       12.1%  3.293e+03   7.6% 
 1:        MG Apply: 3.6228e+01  73.3%  7.3260e+11  94.2%  2.636e+07  98.4%  1.555e+03       87.9%  4.008e+04  92.4% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecNorm              710 1.0 2.7513e-01 4.2 4.65e+07 1.0 0.0e+00 0.0e+00 7.1e+02  0  1  0  0  2   1 13  0  0 22 21647
VecCopy              378 1.0 6.4831e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               160 1.0 2.7622e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY              286 1.0 5.0174e-02 1.2 1.87e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  5  0  0  0 47817
VecAYPX              618 1.0 8.3843e-02 1.9 2.03e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  6  0  0  0 30916
VecScatterBegin      714 1.0 1.3197e-01 1.3 0.00e+00 0.0 3.4e+05 1.6e+04 0.0e+00  0  0  1 12  0   1  0 81 97  0     0
VecScatterEnd        714 1.0 3.0915e-01 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatMult              618 1.0 2.2833e+00 1.1 2.63e+08 1.0 3.2e+05 1.6e+04 0.0e+00  4  4  1 11  0  16 74 75 90  0 14758
MatMultTranspose       4 1.0 2.2891e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0 14132
MatLUFactorSym         1 1.0 5.1403e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 1.1998e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
MatAssemblyBegin      63 1.0 1.9263e-01 5.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02  0  0  0  0  0   1  0  0  0  3     0
MatAssemblyEnd        63 1.0 2.1651e-01 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01  0  0  0  0  0   1  0  3  0  2     0
MatGetRowIJ            1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.0981e-05 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView              690 2.1 2.0276e-01 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.2e+02  0  0  0  0  1   1  0  0  0 10     0
MatPtAP                4 1.0 2.0942e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02  0  0  0  0  0   2  1  6  3  3  3120
MatPtAPSymbolic        4 1.0 1.4736e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01  0  0  0  0  0   1  0  4  2  2     0
MatPtAPNumeric         4 1.0 6.9803e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01  0  0  0  0  0   1  1  2  1  1  9361
MatGetLocalMat         4 1.0 2.3130e-02 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          4 1.0 2.8250e-02 3.0 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00  0  0  0  0  0   0  0  3  2  0     0
MatGetSymTrans         8 1.0 9.6016e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               6 1.0 1.7802e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  0   0  0  0  0  1     0
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
KSPSolve              46 1.0 3.9455e+01 1.0 6.08e+09 1.0 2.7e+07 1.8e+03 4.3e+04 80100100 99 99 299171663218181298 19717
PCSetUp                1 1.0 4.1868e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.3e+02  1  0  0  0  1   3  2  8  3 10  1638
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
PCApply              286 1.0 3.6235e+01 1.0 5.72e+09 1.0 2.6e+07 1.6e+03 4.0e+04 73 94 98 88 92 275161662387251217 20218
MGSetup Level 0        1 1.0 1.2280e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   1  0  0  0  0     0
MGSetup Level 1        1 1.0 2.5809e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 2        1 1.0 5.2381e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 3        1 1.0 5.4312e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 4        1 1.0 1.1581e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: MG Apply

VecMDot            17160 1.0 2.7126e+00 2.0 2.93e+08 1.0 0.0e+00 0.0e+00 1.7e+04  5  5  0  0 40   6  5  0  0 43 13846
VecNorm            22880 1.0 1.5497e+00 1.7 1.96e+08 1.0 0.0e+00 0.0e+00 2.3e+04  2  3  0  0 53   3  3  0  0 57 16159
VecScale           22880 1.0 1.6083e-01 1.1 9.78e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0 77850
VecCopy             7150 1.0 1.9496e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet             41184 1.0 4.9180e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecAXPY             9724 1.0 1.8098e-01 1.3 9.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0 67033
VecAYPX             2860 1.0 5.3519e-02 1.2 1.22e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 29243
VecMAXPY           22880 1.0 4.8374e-01 1.2 4.40e+08 1.0 0.0e+00 0.0e+00 0.0e+00  1  7  0  0  0   1  8  0  0  0 116471
VecScatterBegin    32032 1.0 2.1183e+00 2.1 0.00e+00 0.0 2.6e+07 1.6e+03 0.0e+00  4  0 98 88  0   5  0100100  0     0
VecScatterEnd      32032 1.0 1.8369e+00 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0   4  0  0  0  0     0
VecNormalize       22880 1.0 1.7390e+00 1.6 2.93e+08 1.0 0.0e+00 0.0e+00 2.3e+04  3  5  0  0 53   4  5  0  0 57 21600
MatMult            25168 1.0 1.6588e+01 1.1 2.44e+09 1.0 2.4e+07 1.7e+03 0.0e+00 33 40 89 84  0  44 43 90 96  0 18825
MatMultAdd          2860 1.0 6.8537e-01 1.2 8.25e+07 1.0 1.1e+06 5.3e+02 0.0e+00  1  1  4  1  0   2  1  4  1  0 15414
MatMultTranspose    4004 1.0 1.2716e+00 1.2 1.55e+08 1.0 1.5e+06 6.6e+02 0.0e+00  2  3  6  2  0   3  3  6  2  0 15583
MatSolve           24310 1.0 1.3130e+01 1.1 1.91e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 31  0  0  0  34 33  0  0  0 18627
MatLUFactorNum         4 1.0 1.0485e-02 1.1 1.86e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 22747
MatILUFactorSym        4 1.0 4.9901e-02 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            4 1.0 1.2398e-05 6.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         4 1.0 5.0456e-0224.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog     17160 1.0 3.0344e+00 1.8 5.87e+08 1.0 0.0e+00 0.0e+00 1.7e+04  5 10  0  0 40   7 10  0  0 43 24755
KSPSetUp               4 1.0 5.9605e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            7150 1.0 3.1997e+01 1.0 5.14e+09 1.0 2.1e+07 1.7e+03 4.0e+04 64 85 77 74 92  88 90 78 84100 20558
PCSetUp                4 1.0 7.4455e-02 1.9 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01  0  0  0  0  0   0  0  0  0  0  3203
PCSetUpOnBlocks     5720 1.0 7.9641e-02 1.8 1.86e+06 1.0 0.0e+00 0.0e+00 1.4e+01  0  0  0  0  0   0  0  0  0  0  2995
PCApply            24310 1.0 1.4260e+01 1.1 1.91e+09 1.0 0.0e+00 0.0e+00 0.0e+00 27 31  0  0  0  37 33  0  0  0 17151
MGSmooth Level 0    1430 1.0 2.0421e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  4  0  0  0  0   5  0  0  0  0     0
MGSmooth Level 1    2288 1.0 1.2168e+00 1.1 4.82e+07 1.0 8.5e+06 1.9e+02 1.6e+04  2  1 32  3 37   3  1 32  4 40  5072
MGResid Level 1     1144 1.0 5.2052e-02 1.3 3.95e+06 1.0 1.2e+06 1.9e+02 0.0e+00  0  0  4  0  0   0  0  4  1  0  9722
MGInterp Level 1    2860 1.0 1.5052e-01 3.6 1.22e+06 1.0 1.1e+06 6.4e+01 0.0e+00  0  0  4  0  0   0  0  4  0  0  1035
MGSmooth Level 2    1716 1.0 1.9140e+00 1.0 3.39e+08 1.0 6.4e+06 6.4e+02 1.2e+04  4  6 24  9 28   5  6 24 10 30 22666
MGResid Level 2      858 1.0 1.1919e-01 1.5 2.37e+07 1.0 8.8e+05 6.4e+02 0.0e+00  0  0  3  1  0   0  0  3  1  0 25474
MGInterp Level 2    2288 1.0 1.6201e-01 2.2 7.76e+06 1.0 8.8e+05 2.1e+02 0.0e+00  0  0  3  0  0   0  0  3  0  0  6132
MGSmooth Level 3    1144 1.0 1.0630e+01 1.0 1.98e+09 1.0 4.4e+06 2.3e+03 8.0e+03 21 33 16 21 18  29 35 17 24 20 23810
MGResid Level 3      572 1.0 7.5736e-01 1.1 1.27e+08 1.0 5.9e+05 2.3e+03 0.0e+00  1  2  2  3  0   2  2  2  3  0 21382
MGInterp Level 3    1716 1.0 3.9544e-01 1.4 4.63e+07 1.0 6.6e+05 7.7e+02 0.0e+00  1  1  2  1  0   1  1  2  1  0 14978
MGSmooth Level 4     572 1.0 1.6467e+01 1.0 2.77e+09 1.0 1.2e+06 1.6e+04 4.0e+03 33 46  4 41  9  45 48  4 46 10 21568
MGResid Level 4      286 1.0 1.0729e+00 1.1 1.31e+08 1.0 1.5e+05 1.6e+04 0.0e+00  2  2  1  5  0   3  2  1  6  0 15654
MGInterp Level 4    1144 1.0 1.9588e+00 1.1 2.44e+08 1.0 4.4e+05 2.9e+03 0.0e+00  4  4  2  3  0   5  4  2  3  0 15922
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector   841            853    211053336     0
      Vector Scatter    19             19        22572     0
              Matrix    38             42     19508928     0
   Matrix Null Space     1              1          652     0
    Distributed Mesh     5              5       830792     0
     Bipartite Graph    10             10         8560     0
           Index Set    47             59       844496     0
   IS L to G Mapping     5              5       405756     0
       Krylov Solver    11             11        84080     0
     DMKSP interface     3              3         2088     0
      Preconditioner    11             11        11864     0
              Viewer   185            184       144256     0

--- Event Stage 1: MG Apply

              Vector    12              0            0     0
              Matrix     4              0            0     0
           Index Set    14              2         1792     0
========================================================================================================================
Average time to get PetscTime(): 1.90735e-07
Average time for MPI_Barrier(): 1.37806e-05
Average time for zero size MPI_Send(): 2.67848e-06
#PETSc Option Table entries:
-ksp_atol 1e-9
-ksp_max_it 30
-ksp_monitor_true_residual
-ksp_type richardson
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure run at: Wed Aug 28 23:25:43 2013
Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok  --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
-----------------------------------------
Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 
Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2
Using PETSc arch: interlagos-64idx-pgi-opt
-----------------------------------------

Using C compiler: cc  -O3 -fastsse  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ftn  -O3 -fastsse   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include
-----------------------------------------

Using C linker: cc
Using Fortran linker: ftn
Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl 
-----------------------------------------

#PETSc Option Table entries:
-ksp_atol 1e-9
-ksp_max_it 30
-ksp_monitor_true_residual
-ksp_type richardson
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type gmres
-mg_levels_pc_type bjacobi
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_mg_type full
-pc_type mg
#End of PETSc Option Table entries
There are no unused options.

From bsmith at mcs.anl.gov  Tue Oct  1 21:22:08 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 Oct 2013 21:22:08 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524B7EB4.8040108@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
Message-ID: <9B4CB541-DC44-4194-A7F4-B2D0DD1032A2@mcs.anl.gov>


  Thanks. Unacceptably poor convergence of multigrid here. 

  What kind of discretization are you using for your "projection method"? Is is cell centered? What kind of boundary conditions? 

  If you are using cell-centered (piece-wise constant) then the interpolation you are getting is all wrong for your problem if you are using the piece-wise linear interpolation. Src/ksp/ksp/examples/tutorials/ex34.c note the call to   ierr = DMDASetInterpolationType(da, DMDA_Q0);CHKERRQ(ierr);

   Barry



On Oct 1, 2013, at 9:02 PM, Michele Rosso <mrosso at uci.edu> wrote:

> Barry,
> 
> I repeated the previous runs since I noticed that it was not using the options
> 
> -mg_levels_ksp_max_it 3
> 
> They are faster then before now but still slower than my initial test and anyway the solution time increases considerably in time.
> I attached the diagnostics (run1.txt and run2.txt, please see the files for the list of the options I used).
> I also run a case by using your last proposed options (run3.txt): there is a divergence condition since 30 iterations seem not be enough to low the error below the need tolerance and thus after some time steps my solution blows up.
> Please let me know what you think about it.
> In the mean time I will try run my initial test with the option -mg_levels_ksp_max_it 3 instead of  -mg_levels_ksp_max_it 1.
> As usual, thank you very much.
> 
> Michele
> 
> On 09/30/2013 07:17 PM, Barry Smith wrote:
>>   I wasn't expecting this. Try
>> 
>> -pc_mg_type full -ksp_type richardson -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3
>> -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg
>> -log_summary  -pc_mg_log  -ksp_monitor_true_residual  -options_left   -ksp_view -ksp_max_it 30
>> 
>> 
>> 
>> 
>> On Sep 30, 2013, at 8:00 PM, Michele Rosso <mrosso at uci.edu> wrote:
>> 
>>> Barry,
>>> 
>>> sorry again for the very late answer. I tried all the variations you proposed: all of them converge very slow , except the last one (CG instead of fgmres) diverges.
>>> I attached the diagnostics for the two options that convergence: each of the attached files starts with a list of the options I used for the run.
>>> As you pointed out earlier, the residual I used to require was too small, therefore I increased atol to e-9.
>>> After some tests, I noticed that any further increase of the absolute tolerance changes significantly the solution.
>>> What would you suggest to try next?
>>> Thank you very much,
>>> 
>>> Michele
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 09/24/2013 05:08 PM, Barry Smith wrote:
>>>>  Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate.
>>>> 
>>>>  The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ????
>>>> 
>>>>  Start with a really robust multigrid smoother use
>>>> 
>>>> -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3   PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>> -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg
>>>> 
>>>>  run with the -log_summary and -pc_mg_log
>>>> 
>>>> Now back off a little on the smoother and use -mg_levels_pc_type sor instead  how does that change the convergence and time.
>>>> 
>>>> Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson   how does that change the convergence and the time?
>>>> 
>>>>  There are some additional variants we can try based on the results from above.
>>>> 
>>>>  Barry
>>>> 
>>>> 
>>>> 
>>>> On Sep 24, 2013, at 4:29 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>>> 
>>>>> Barry,
>>>>> 
>>>>> I re-rerun the test case with the option -pc_mg_log as you suggested.
>>>>> I attached the new output ("final_new.txt').
>>>>> Thanks for your help.
>>>>> 
>>>>> Michele
>>>>> 
>>>>> On 09/23/2013 09:35 AM, Barry Smith wrote:
>>>>>>   Run with the additional option -pc_mg_log and send us the log file.
>>>>>> 
>>>>>>   Barry
>>>>>> 
>>>>>> Maybe we should make this the default somehow.
>>>>>> 
>>>>>> 
>>>>>> On Sep 23, 2013, at 10:55 AM, Michele Rosso <mrosso at uci.edu> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG .  Such equation arises from a projection algorithm for a multiphase incompressible flow simulation.
>>>>>>> I set up the solver  as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid).
>>>>>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations.
>>>>>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can.
>>>>>>> Could you please help me out with this?
>>>>>>> 
>>>>>>> I run the test case with the following options:
>>>>>>> 
>>>>>>> -pc_type mg  -pc_mg_galerkin  -pc_mg_levels 5   -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
>>>>>>> -mg_coarse_pc_type lu   -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>>>>> -log_summary -ksp_view  -ksp_monitor_true_residual  -options_left
>>>>>>> 
>>>>>>> Please find the diagnostic for the final solve in the attached file "final.txt'.
>>>>>>> Thank you,
>>>>>>> 
>>>>>>> Michele
>>>>>>> <final.txt>
>>>>> <final_new.txt>
>>> <final1.txt><final2.txt>
>> 
> 
> <run1.txt><run2.txt><run3.txt>


From gryllida at fastmail.fm  Tue Oct  1 23:03:12 2013
From: gryllida at fastmail.fm (Gryllida)
Date: Wed, 02 Oct 2013 13:33:12 +0930
Subject: [petsc-users] using mpi
Message-ID: <1380686592.31832.28925085.7298F3F1@webmail.messagingengine.com>

I have a parallel program which reads data and makes a call to a subroutine that uses petsc. It only needs to read data once. Is it possible to implement that using MPI and then initialise petsc in the subroutine? Or do I need to initialise petsc first (and use MPI calls with PETSC_COMM_WORLD)?

From jedbrown at mcs.anl.gov  Tue Oct  1 23:08:46 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 01 Oct 2013 23:08:46 -0500
Subject: [petsc-users] using mpi
In-Reply-To: <1380686592.31832.28925085.7298F3F1@webmail.messagingengine.com>
References: <1380686592.31832.28925085.7298F3F1@webmail.messagingengine.com>
Message-ID: <87zjqsz54h.fsf@mcs.anl.gov>

Gryllida <gryllida at fastmail.fm> writes:

> I have a parallel program which reads data and makes a call to a
> subroutine that uses petsc. It only needs to read data once. Is it
> possible to implement that using MPI and then initialise petsc in the
> subroutine? Or do I need to initialise petsc first (and use MPI calls
> with PETSC_COMM_WORLD)?

We recommend only calling PetscInitialize and PetscFinalize once, so you
may want to lift it to an initialization function.  You can call them
inside the function, but if the function is called multiple times, the
profiling and debugging functionality in PETSc will have to run each
time, producing more output than you probably want when you turn those
features on.

Regardless of where you initialize PETSc, you can use MPI_COMM_WORLD or
any other communicator.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131001/47ef5838/attachment.pgp>

From mfadams at lbl.gov  Tue Oct  1 23:49:18 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Tue, 1 Oct 2013 21:49:18 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524B7EB4.8040108@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
Message-ID: <D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>


On Oct 1, 2013, at 7:02 PM, Michele Rosso <mrosso at uci.edu> wrote:

> Barry,
> 
> I repeated the previous runs since I noticed that it was not using the options
> 
> -mg_levels_ksp_max_it 3
> 
> They are faster then before now but still slower than my initial test and anyway the solution time increases considerably in time.
> I attached the diagnostics (run1.txt and run2.txt, please see the files for the list of the options I used).
> I also run a case by using your last proposed options (run3.txt): there is a divergence condition since 30 iterations seem not be enough to low the error below the need tolerance and thus after some time steps my solution blows up.

run3.txt uses:

-ksp_type richardson

This is bad and I doubt anyone recommended it intentionally.  You also have, in this file,

-mg_levels_ksp_type gmres

did you or the recommenders mean

-mg_levels_ksp_type richardson  ???

you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then

-ksp_type cg
-mg_levels_ksp_type richardson 
-mg_levels_pc_type sor

is what I'd recommend.  

Your run2.txt looks like the best.  If I understand you you have better results with some other parameters ("initial test").  

From what I can see you are solving a Laplacia with constant coefficients?

I would also add Barry's question to what we want to know.

Mark


> Please let me know what you think about it.
> In the mean time I will try run my initial test with the option -mg_levels_ksp_max_it 3 instead of  -mg_levels_ksp_max_it 1.
> As usual, thank you very much.
> 
> Michele
> 
> On 09/30/2013 07:17 PM, Barry Smith wrote:
>>   I wasn't expecting this. Try
>> 
>> -pc_mg_type full -ksp_type richardson -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3
>> -mg_coarse_pc_factor_mat_solver_package superlu_dist -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_type mg
>> -log_summary  -pc_mg_log  -ksp_monitor_true_residual  -options_left   -ksp_view -ksp_max_it 30
>> 
>> 
>> 
>> 
>> On Sep 30, 2013, at 8:00 PM, Michele Rosso <mrosso at uci.edu> wrote:
>> 
>>> Barry,
>>> 
>>> sorry again for the very late answer. I tried all the variations you proposed: all of them converge very slow , except the last one (CG instead of fgmres) diverges.
>>> I attached the diagnostics for the two options that convergence: each of the attached files starts with a list of the options I used for the run.
>>> As you pointed out earlier, the residual I used to require was too small, therefore I increased atol to e-9.
>>> After some tests, I noticed that any further increase of the absolute tolerance changes significantly the solution.
>>> What would you suggest to try next?
>>> Thank you very much,
>>> 
>>> Michele
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 09/24/2013 05:08 PM, Barry Smith wrote:
>>>>  Thanks. The balance of work on the different levels and across processes looks ok. So it is is a matter of improving the convergence rate.
>>>> 
>>>>  The initial residual norm is very small. Are you sure you need to decrease it to 10^-12 ????
>>>> 
>>>>  Start with a really robust multigrid smoother use
>>>> 
>>>> -pc_mg_type full -ksp_type fgmres -mg_levels_pc_type bjacobi -mg_levels_ksp_type gmres -mg_levels_ksp_max_it 3   PLUS -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>> -mg_coarse_pc_type lu -pc_mg_galerkin -pc_mg_levels 5 -pc_mg_log -pc_type mg
>>>> 
>>>>  run with the -log_summary and -pc_mg_log
>>>> 
>>>> Now back off a little on the smoother and use -mg_levels_pc_type sor instead  how does that change the convergence and time.
>>>> 
>>>> Back off even more an replace the -ksp_type fgmres with -ksp_type cg and the -mg_levels_ksp_type gmres with -mg_levels_ksp_type richardson   how does that change the convergence and the time?
>>>> 
>>>>  There are some additional variants we can try based on the results from above.
>>>> 
>>>>  Barry
>>>> 
>>>> 
>>>> 
>>>> On Sep 24, 2013, at 4:29 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>>> 
>>>>> Barry,
>>>>> 
>>>>> I re-rerun the test case with the option -pc_mg_log as you suggested.
>>>>> I attached the new output ("final_new.txt').
>>>>> Thanks for your help.
>>>>> 
>>>>> Michele
>>>>> 
>>>>> On 09/23/2013 09:35 AM, Barry Smith wrote:
>>>>>>   Run with the additional option -pc_mg_log and send us the log file.
>>>>>> 
>>>>>>   Barry
>>>>>> 
>>>>>> Maybe we should make this the default somehow.
>>>>>> 
>>>>>> 
>>>>>> On Sep 23, 2013, at 10:55 AM, Michele Rosso <mrosso at uci.edu> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I am successfully using PETSc to solve a 3D Poisson's equation with CG + MG .  Such equation arises from a projection algorithm for a multiphase incompressible flow simulation.
>>>>>>> I set up the solver  as I was suggested to do in a previous thread (title: "GAMG speed") and run a test case (liquid droplet with surface tension falling under the effect of gravity in a quiescent fluid).
>>>>>>> The solution of the Poisson Equation via multigrid is correct but it becomes progressively slower and slower as the simulation progresses (I am performing successive solves) due to an increase in the number of iterations.
>>>>>>> Since the solution of the Poisson equation is mission-critical, I need to speed it up as much as I can.
>>>>>>> Could you please help me out with this?
>>>>>>> 
>>>>>>> I run the test case with the following options:
>>>>>>> 
>>>>>>> -pc_type mg  -pc_mg_galerkin  -pc_mg_levels 5   -mg_levels_ksp_type richardson -mg_levels_ksp_max_it 1
>>>>>>> -mg_coarse_pc_type lu   -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>>>>> -log_summary -ksp_view  -ksp_monitor_true_residual  -options_left
>>>>>>> 
>>>>>>> Please find the diagnostic for the final solve in the attached file "final.txt'.
>>>>>>> Thank you,
>>>>>>> 
>>>>>>> Michele
>>>>>>> <final.txt>
>>>>> <final_new.txt>
>>> <final1.txt><final2.txt>
>> 
> 
> <run1.txt><run2.txt><run3.txt>


From gryllida at fastmail.fm  Tue Oct  1 23:59:56 2013
From: gryllida at fastmail.fm (Gryllida)
Date: Wed, 02 Oct 2013 14:29:56 +0930
Subject: [petsc-users] using mpi
In-Reply-To: <87zjqsz54h.fsf@mcs.anl.gov>
References: <1380686592.31832.28925085.7298F3F1@webmail.messagingengine.com>
	<87zjqsz54h.fsf@mcs.anl.gov>
Message-ID: <1380689996.13560.28935597.2AF09D00@webmail.messagingengine.com>

On Wed, 2 Oct 2013, at 13:38, Jed Brown wrote:
> Gryllida <gryllida at fastmail.fm> writes:
> 
> > I have a parallel program which reads data and makes a call to a
> > subroutine that uses petsc. It only needs to read data once. Is it
> > possible to implement that using MPI and then initialise petsc in the
> > subroutine? Or do I need to initialise petsc first (and use MPI calls
> > with PETSC_COMM_WORLD)?
> 
> We recommend only calling PetscInitialize and PetscFinalize once, so you
> may want to lift it to an initialization function.  You can call them
> inside the function, but if the function is called multiple times, the
> profiling and debugging functionality in PETSc will have to run each
> time, producing more output than you probably want when you turn those
> features on.
> 
> Regardless of where you initialize PETSc, you can use MPI_COMM_WORLD or
> any other communicator.

Thank you for your nudge toward that using MPI first is possible; I had a system-wide MPI installation but did not specify it to ./configure. Doing so resolved my problems.

From balay at mcs.anl.gov  Wed Oct  2 00:07:54 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 2 Oct 2013 00:07:54 -0500 (CDT)
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <CALmcJSCsvmyUdx_RjdSLqbXt_5D7oosaJk02M=6a_2MOOcsdHw@mail.gmail.com>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
	<alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
	<CALmcJSCsvmyUdx_RjdSLqbXt_5D7oosaJk02M=6a_2MOOcsdHw@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310020004130.1897@mcs.anl.gov>

On Tue, 1 Oct 2013, Mengda Wu wrote:

> Hi Satish,
> 
>    Thanks a lot for your reply!
> 
>    Is there a way to use the mingw64 which is not shipped with cygwin? For
> example, use windows native paths when testing the compiler.

Nope - I don't think it will work. Our build tools are primarily work with
cygwin paths.

Alternative is to use MS/Intel compilers - and we support them with
win32fe compiler wrapper.

What problem are you having with using mingw distributed with cygwin?

Satish

From mirzadeh at gmail.com  Wed Oct  2 00:27:01 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 1 Oct 2013 22:27:01 -0700
Subject: [petsc-users] ISLocalToGlobalMapping + VecCreateGhost
Message-ID: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>

Hi guys,

I just did something by pure guessing which seems to work and I want
to make sure its the right thing!

I have a specific layout for my vectors that look like this

 --------------------------------------------------------------
| ghost values | local values | ghost values |
 --------------------------------------------------------------
0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N

which means all nodes with index [0,m) and [m+n, N) are to be treated
as ghost and all intermediate ones as local. Since PETSc stores the
ghost values at the end of ghosted vec, so far I have been manually
mapping back and forth between petsc and my application numbering.
After profiling my code, it turned out that about 15% of run-time was
just doing this mapping which is absolutely ridiculous!

Anyway, to fix this now I set up an ISLocalToGlobalMapping object
using my own application original local2global relation shown above.
Then I create the petsc vec like this:

// Create PetscVec
// We do not care about the actual global index of ghost nodes at this point
std::vector<PetscInt> ghost_nodes(N - n -1, 0);
ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
(const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);


// Apply mapping
ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);

After this point I do the usual VecGetArray on the vec and set the
values, but this time without any extra mapping ... my very simple
test seems to be ok. Is this the correct way of using
ISLocalToGlobalMapping?

I guess I'm suspicious because VecCreateGhost is supposed to
internally set the mapping and it is supposed to position ghost nodes
at the end of the array which I don't want it to do...

Thanks and sorry about the long email!

From jedbrown at mcs.anl.gov  Wed Oct  2 00:28:25 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 02 Oct 2013 00:28:25 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
Message-ID: <87pproz1fq.fsf@mcs.anl.gov>

"Mark F. Adams" <mfadams at lbl.gov> writes:
> run3.txt uses:
>
> -ksp_type richardson
>
> This is bad and I doubt anyone recommended it intentionally.  

I would have expected FGMRES, but Barry likes Krylov smoothers and
Richardson is one of a few methods that can tolerate nonlinear
preconditioners.

> You also have, in this file,
>
> -mg_levels_ksp_type gmres
>
> did you or the recommenders mean
>
> -mg_levels_ksp_type richardson  ???
>
> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>
> -ksp_type cg
> -mg_levels_ksp_type richardson 
> -mg_levels_pc_type sor
>
> is what I'd recommend.  

I thought that was tried in an earlier round.

I don't understand why SOR preconditioning in the Krylov smoother is so
drastically more expensive than BJacobi/ILU and why SOR is called so
many more times even though the number of outer iterations 

bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800

sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/2c61bdc1/attachment.pgp>

From bsmith at mcs.anl.gov  Wed Oct  2 01:44:41 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 Oct 2013 01:44:41 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <87pproz1fq.fsf@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
Message-ID: <B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>


On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> "Mark F. Adams" <mfadams at lbl.gov> writes:
>> run3.txt uses:
>> 
>> -ksp_type richardson
>> 
>> This is bad and I doubt anyone recommended it intentionally.  

   Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice. 

> 
> I would have expected FGMRES, but Barry likes Krylov smoothers and
> Richardson is one of a few methods that can tolerate nonlinear
> preconditioners.
> 
>> You also have, in this file,
>> 
>> -mg_levels_ksp_type gmres
>> 
>> did you or the recommenders mean
>> 
>> -mg_levels_ksp_type richardson  ???
>> 
>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>> 
>> -ksp_type cg
>> -mg_levels_ksp_type richardson 
>> -mg_levels_pc_type sor
>> 
>> is what I'd recommend.  
> 
> I thought that was tried in an earlier round.
> 
> I don't understand why SOR preconditioning in the Krylov smoother is so
> drastically more expensive than BJacobi/ILU and why SOR is called so
> many more times even though the number of outer iterations 
> 
> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
> 
> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652


From knepley at gmail.com  Wed Oct  2 06:13:48 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 Oct 2013 06:13:48 -0500
Subject: [petsc-users] ISLocalToGlobalMapping + VecCreateGhost
In-Reply-To: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>
References: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>
Message-ID: <CAMYG4G=7buZxL7hdmNoX_rOxWq-wwoTagDhCZD1b-RA6yZNsdA@mail.gmail.com>

On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi guys,
>
> I just did something by pure guessing which seems to work and I want
> to make sure its the right thing!
>
> I have a specific layout for my vectors that look like this
>
>  --------------------------------------------------------------
> | ghost values | local values | ghost values |
>  --------------------------------------------------------------
> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
>
> which means all nodes with index [0,m) and [m+n, N) are to be treated
> as ghost and all intermediate ones as local. Since PETSc stores the
> ghost values at the end of ghosted vec, so far I have been manually
> mapping back and forth between petsc and my application numbering.
> After profiling my code, it turned out that about 15% of run-time was
> just doing this mapping which is absolutely ridiculous!
>
> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
> using my own application original local2global relation shown above.
> Then I create the petsc vec like this:
>
> // Create PetscVec
> // We do not care about the actual global index of ghost nodes at this
> point
> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
>
>
> // Apply mapping
> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
>
> After this point I do the usual VecGetArray on the vec and set the
> values, but this time without any extra mapping ... my very simple
> test seems to be ok. Is this the correct way of using
> ISLocalToGlobalMapping?
>

Yes, this is the intention. And at a higher level, PETSc now tends to use a
DM to
organize this process. You tell the DM about your data layout (somewhat
like giving
the L2G mapping) and then you can DMGetLocalVector(), DMGetGlobalVector(),
and
DMLocalToGlobal().

  Thanks,

     Matt


> I guess I'm suspicious because VecCreateGhost is supposed to
> internally set the mapping and it is supposed to position ghost nodes
> at the end of the array which I don't want it to do...
>
> Thanks and sorry about the long email!
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/aa9b88cb/attachment.html>

From knepley at gmail.com  Wed Oct  2 07:39:39 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 Oct 2013 07:39:39 -0500
Subject: [petsc-users] Out of memory while using
	/src/src/ksp/example/tutorial/ex45.c
In-Reply-To: <5249FDCE.4070503@gmail.com>
References: <5249FDCE.4070503@gmail.com>
Message-ID: <CAMYG4G=Ew3VQdWvJ-CQn0QcmfWWMZwzsxEMWo=x+gVo0JeNa0Q@mail.gmail.com>

On Mon, Sep 30, 2013 at 5:40 PM, Alan Z. Wei <zhenglun.wei at gmail.com> wrote:

> Dear all,
> I hope you're having a nice day.
> I came across a memory problem while using ex45.c to solve 3D Poisson
> equation. As recommended while using 2D Poisson solver (e.g
> /src/src/ksp/example/tutorial/ex29.c), I used the multigrid
> preconditioner and the setting is listed here:
> mpiexec -np 8 ./ex45 -pc_type mg -ksp_type cg -da_refine 2
> -pc_mg_galerkin -ksp_rtol 1.0e-7 -mg_levels_pc_type jacobi
> -mg_levels_ksp_type chebyshev
> If I apply the same settings to ex45 with np = 32 and it results a 'out
> of memory' error. The case has grid size = 801*385*385. Of course, if I
>

ALWAYS send the full error text.


> reduced the grid size to very small, it would work well. For the test
> case, I tried to play with
> 1, doubleing the number of process (by using np =64, total avaiable
> memory should be aroud 128G)
> 2, reducing the grid size to half
> 3, reducing the level of the MG to 1
> They do not help at all. What should I output in order to detect the
> problem?
>

This is a very stripped down example, so the only memory should be in the
sol+res vectors, Jacobian, DMDA, and Krylov solver. Do you have estimates
for these? Have you looked at what -log_summary reports?

  Thanks,

      Matt


> BTW, the ex45 works well with the same amount of DoF with GAMG and the
> following settings however it is slow,
> mpiexec -np 32 ./ex45 -pc_type gamg -ksp_type cg -pc_gamg_agg_nsmooths 1
> -mg_levels_ksp_max_it 1 -mg_levels_ksp_type richardson -ksp_rtol 1.0e-7.
> I understand that MG requires more memory than GAMG; yet, is there any
> way I can approximately calculate the memory required?
>
> Thanks in advance,
> Alan
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/b5b3fddd/attachment.html>

From mailinglistxf at yahoo.fr  Wed Oct  2 07:59:20 2013
From: mailinglistxf at yahoo.fr (mailinglist)
Date: Wed, 02 Oct 2013 14:59:20 +0200
Subject: [petsc-users] VecRestoreSubVector can't change value of original
	vector
Message-ID: <524C18A8.5050609@yahoo.fr>

Hello,

I have a vector(v). I use VecRestoreSubVector to get a sub vector(subv). 
I change the value of the sub vector and want to give the new value back 
to v using VecRestoreSubVector, but it only works for seqential 
case(mpirun -np 1 ...). This is my code. Thank you very much.

#include "petsc.h"
#include "mpi.h"

int main(int argc, char** argv){

   PetscInitialize(&argc,&argv,0,0);

   int commSize;
   int commRank;
   MPI_Comm_rank(MPI_COMM_WORLD,&commRank);
   MPI_Comm_size(MPI_COMM_WORLD,&commSize);

   Vec v;
   VecCreateMPI(PETSC_COMM_WORLD,5,5*commSize,&v);
   VecSet(v,0.0);
   VecAssemblyBegin(v);
   VecAssemblyEnd(v);

   // is
   IS is;
   int index[1], Nindex = 1;
   index[0] = 0;
ISCreateGeneral(PETSC_COMM_WORLD,Nindex,index,PETSC_COPY_VALUES,&is);

   Vec vsub;
   VecGetSubVector(v,is,&vsub);
   VecSet(vsub,1);
   VecAssemblyBegin(vsub);
   VecAssemblyEnd(vsub);

   // VecView(vsub,PETSC_VIEWER_STDOUT_WORLD);

   VecRestoreSubVector(v,is,&vsub);
   // VecAssemblyBegin(v);
   // VecAssemblyEnd(v);

   VecView(v,PETSC_VIEWER_STDOUT_WORLD);

   PetscFinalize();

   return 0;
}

Regards,

fxing

From jedbrown at mcs.anl.gov  Wed Oct  2 08:04:56 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 02 Oct 2013 08:04:56 -0500
Subject: [petsc-users] VecRestoreSubVector can't change value of
	original vector
In-Reply-To: <524C18A8.5050609@yahoo.fr>
References: <524C18A8.5050609@yahoo.fr>
Message-ID: <871u43zuvb.fsf@mcs.anl.gov>

mailinglist <mailinglistxf at yahoo.fr> writes:

> Hello,
>
> I have a vector(v). I use VecRestoreSubVector to get a sub vector(subv). 
> I change the value of the sub vector and want to give the new value back 
> to v using VecRestoreSubVector, but it only works for seqential 
> case(mpirun -np 1 ...). 

The interface is supposed to give you read access, but not currently
write access.  The interface doesn't say whether the values are copied.
Your subvector is not contiguous so it can't be done "in-place".  You
should probably use VecScatter for this purpose.

> This is my code. Thank you very much.
>
> #include "petsc.h"
> #include "mpi.h"
>
> int main(int argc, char** argv){
>
>    PetscInitialize(&argc,&argv,0,0);
>
>    int commSize;
>    int commRank;
>    MPI_Comm_rank(MPI_COMM_WORLD,&commRank);
>    MPI_Comm_size(MPI_COMM_WORLD,&commSize);
>
>    Vec v;
>    VecCreateMPI(PETSC_COMM_WORLD,5,5*commSize,&v);
>    VecSet(v,0.0);
>    VecAssemblyBegin(v);
>    VecAssemblyEnd(v);
>
>    // is
>    IS is;
>    int index[1], Nindex = 1;
>    index[0] = 0;
> ISCreateGeneral(PETSC_COMM_WORLD,Nindex,index,PETSC_COPY_VALUES,&is);
>
>    Vec vsub;
>    VecGetSubVector(v,is,&vsub);
>    VecSet(vsub,1);
>    VecAssemblyBegin(vsub);
>    VecAssemblyEnd(vsub);
>
>    // VecView(vsub,PETSC_VIEWER_STDOUT_WORLD);
>
>    VecRestoreSubVector(v,is,&vsub);
>    // VecAssemblyBegin(v);
>    // VecAssemblyEnd(v);
>
>    VecView(v,PETSC_VIEWER_STDOUT_WORLD);
>
>    PetscFinalize();
>
>    return 0;
> }
>
> Regards,
>
> fxing
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/27abda5a/attachment-0001.pgp>

From mrosso at uci.edu  Wed Oct  2 13:10:32 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Wed, 02 Oct 2013 11:10:32 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
Message-ID: <524C6198.4000506@uci.edu>

Thank you all for your contribution.
So far the fastest solution is still the initial one proposed by Jed in 
an earlier round:

-ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary 
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type 
richardson  -options_left -pc_mg_galerkin
-pc_mg_levels 5  -pc_mg_log  -pc_type mg

where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  
-mg_levels_ksp_max_it 1.
I attached the diagnostics for this case. Any further idea?
Thank you,

Michele


On 10/01/2013 11:44 PM, Barry Smith wrote:
> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> "Mark F. Adams" <mfadams at lbl.gov> writes:
>>> run3.txt uses:
>>>
>>> -ksp_type richardson
>>>
>>> This is bad and I doubt anyone recommended it intentionally.
>     Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>
>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>> Richardson is one of a few methods that can tolerate nonlinear
>> preconditioners.
>>
>>> You also have, in this file,
>>>
>>> -mg_levels_ksp_type gmres
>>>
>>> did you or the recommenders mean
>>>
>>> -mg_levels_ksp_type richardson  ???
>>>
>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>
>>> -ksp_type cg
>>> -mg_levels_ksp_type richardson
>>> -mg_levels_pc_type sor
>>>
>>> is what I'd recommend.
>> I thought that was tried in an earlier round.
>>
>> I don't understand why SOR preconditioning in the Krylov smoother is so
>> drastically more expensive than BJacobi/ILU and why SOR is called so
>> many more times even though the number of outer iterations
>>
>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>
>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>

-------------- next part --------------
OPTIONS USED:
-ksp_atol 1e-9
-ksp_monitor_true_residual
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type richardson
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_type mg

  0 KSP unpreconditioned resid norm 1.062110078078e-06 true resid norm 1.062110078078e-06 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 2.236451946298e-07 true resid norm 2.236451946298e-07 ||r(i)||/||b|| 2.105668698998e-01
  2 KSP unpreconditioned resid norm 1.220833343347e-07 true resid norm 1.220833343347e-07 ||r(i)||/||b|| 1.149441445426e-01
  3 KSP unpreconditioned resid norm 8.872102504003e-08 true resid norm 8.872102504003e-08 ||r(i)||/||b|| 8.353279652576e-02
  4 KSP unpreconditioned resid norm 6.619711859327e-08 true resid norm 6.619711859327e-08 ||r(i)||/||b|| 6.232604318479e-02
  5 KSP unpreconditioned resid norm 5.271840072756e-08 true resid norm 5.271840072756e-08 ||r(i)||/||b|| 4.963553384502e-02
  6 KSP unpreconditioned resid norm 4.229219228929e-08 true resid norm 4.229219228929e-08 ||r(i)||/||b|| 3.981902927221e-02
  7 KSP unpreconditioned resid norm 3.634611900984e-08 true resid norm 3.634611900984e-08 ||r(i)||/||b|| 3.422067049361e-02
  8 KSP unpreconditioned resid norm 3.131796519117e-08 true resid norm 3.131796519117e-08 ||r(i)||/||b|| 2.948655307729e-02
  9 KSP unpreconditioned resid norm 2.653911712772e-08 true resid norm 2.653911712772e-08 ||r(i)||/||b|| 2.498716251309e-02
 10 KSP unpreconditioned resid norm 2.451248019105e-08 true resid norm 2.451248019105e-08 ||r(i)||/||b|| 2.307903926061e-02
 11 KSP unpreconditioned resid norm 2.373835128699e-08 true resid norm 2.373835128699e-08 ||r(i)||/||b|| 2.235017987020e-02
 12 KSP unpreconditioned resid norm 2.476463969748e-08 true resid norm 2.476463969748e-08 ||r(i)||/||b|| 2.331645298226e-02
 13 KSP unpreconditioned resid norm 2.689648556959e-08 true resid norm 2.689648556959e-08 ||r(i)||/||b|| 2.532363276156e-02
 14 KSP unpreconditioned resid norm 3.098937516975e-08 true resid norm 3.098937516975e-08 ||r(i)||/||b|| 2.917717834468e-02
 15 KSP unpreconditioned resid norm 3.555038182753e-08 true resid norm 3.555038182753e-08 ||r(i)||/||b|| 3.347146643395e-02
 16 KSP unpreconditioned resid norm 4.112024511716e-08 true resid norm 4.112024511716e-08 ||r(i)||/||b|| 3.871561523226e-02
 17 KSP unpreconditioned resid norm 4.629240103388e-08 true resid norm 4.629240103388e-08 ||r(i)||/||b|| 4.358531379127e-02
 18 KSP unpreconditioned resid norm 4.985610207288e-08 true resid norm 4.985610207288e-08 ||r(i)||/||b|| 4.694061670436e-02
 19 KSP unpreconditioned resid norm 5.046376291690e-08 true resid norm 5.046376291690e-08 ||r(i)||/||b|| 4.751274275472e-02
 20 KSP unpreconditioned resid norm 5.025357388082e-08 true resid norm 5.025357388082e-08 ||r(i)||/||b|| 4.731484515407e-02
 21 KSP unpreconditioned resid norm 4.733061043311e-08 true resid norm 4.733061043311e-08 ||r(i)||/||b|| 4.456281077642e-02
 22 KSP unpreconditioned resid norm 4.482409805557e-08 true resid norm 4.482409805557e-08 ||r(i)||/||b|| 4.220287424134e-02
 23 KSP unpreconditioned resid norm 4.070552710576e-08 true resid norm 4.070552710576e-08 ||r(i)||/||b|| 3.832514910266e-02
 24 KSP unpreconditioned resid norm 3.746139586173e-08 true resid norm 3.746139586173e-08 ||r(i)||/||b|| 3.527072818058e-02
 25 KSP unpreconditioned resid norm 3.416470090249e-08 true resid norm 3.416470090249e-08 ||r(i)||/||b|| 3.216681736447e-02
 26 KSP unpreconditioned resid norm 3.162747159737e-08 true resid norm 3.162747159737e-08 ||r(i)||/||b|| 2.977796016644e-02
 27 KSP unpreconditioned resid norm 2.886965691540e-08 true resid norm 2.886965691540e-08 ||r(i)||/||b|| 2.718141698425e-02
 28 KSP unpreconditioned resid norm 2.669294602696e-08 true resid norm 2.669294602696e-08 ||r(i)||/||b|| 2.513199580525e-02
 29 KSP unpreconditioned resid norm 2.477496636609e-08 true resid norm 2.477496636609e-08 ||r(i)||/||b|| 2.332617576789e-02
 30 KSP unpreconditioned resid norm 2.254756345946e-08 true resid norm 2.254756345946e-08 ||r(i)||/||b|| 2.122902693878e-02
 31 KSP unpreconditioned resid norm 2.100745862543e-08 true resid norm 2.100745862543e-08 ||r(i)||/||b|| 1.977898436239e-02
 32 KSP unpreconditioned resid norm 2.082372673705e-08 true resid norm 2.082372673705e-08 ||r(i)||/||b|| 1.960599674823e-02
 33 KSP unpreconditioned resid norm 2.058561394284e-08 true resid norm 2.058561394284e-08 ||r(i)||/||b|| 1.938180831510e-02
 34 KSP unpreconditioned resid norm 2.071527481693e-08 true resid norm 2.071527481693e-08 ||r(i)||/||b|| 1.950388688000e-02
 35 KSP unpreconditioned resid norm 2.100892944872e-08 true resid norm 2.100892944872e-08 ||r(i)||/||b|| 1.978036917487e-02
 36 KSP unpreconditioned resid norm 2.220101872142e-08 true resid norm 2.220101872142e-08 ||r(i)||/||b|| 2.090274744554e-02
 37 KSP unpreconditioned resid norm 2.324772438230e-08 true resid norm 2.324772438230e-08 ||r(i)||/||b|| 2.188824384792e-02
 38 KSP unpreconditioned resid norm 2.452302256995e-08 true resid norm 2.452302256995e-08 ||r(i)||/||b|| 2.308896514224e-02
 39 KSP unpreconditioned resid norm 2.502647686575e-08 true resid norm 2.502647686575e-08 ||r(i)||/||b|| 2.356297843539e-02
 40 KSP unpreconditioned resid norm 2.531223073672e-08 true resid norm 2.531223073672e-08 ||r(i)||/||b|| 2.383202199016e-02
 41 KSP unpreconditioned resid norm 2.499727165695e-08 true resid norm 2.499727165695e-08 ||r(i)||/||b|| 2.353548108892e-02
 42 KSP unpreconditioned resid norm 2.462083389942e-08 true resid norm 2.462083389942e-08 ||r(i)||/||b|| 2.318105666033e-02
 43 KSP unpreconditioned resid norm 2.360189108305e-08 true resid norm 2.360189108305e-08 ||r(i)||/||b|| 2.222169958670e-02
 44 KSP unpreconditioned resid norm 2.252988454814e-08 true resid norm 2.252988454814e-08 ||r(i)||/||b|| 2.121238185492e-02
 45 KSP unpreconditioned resid norm 2.188564712770e-08 true resid norm 2.188564712770e-08 ||r(i)||/||b|| 2.060581815334e-02
 46 KSP unpreconditioned resid norm 2.002949813700e-08 true resid norm 2.002949813700e-08 ||r(i)||/||b|| 1.885821305193e-02
 47 KSP unpreconditioned resid norm 1.822159592332e-08 true resid norm 1.822159592332e-08 ||r(i)||/||b|| 1.715603335231e-02
 48 KSP unpreconditioned resid norm 1.731437653543e-08 true resid norm 1.731437653543e-08 ||r(i)||/||b|| 1.630186634399e-02
 49 KSP unpreconditioned resid norm 1.582438316044e-08 true resid norm 1.582438316044e-08 ||r(i)||/||b|| 1.489900480850e-02
 50 KSP unpreconditioned resid norm 1.470070282545e-08 true resid norm 1.470070282545e-08 ||r(i)||/||b|| 1.384103505736e-02
 51 KSP unpreconditioned resid norm 1.317055921275e-08 true resid norm 1.317055921275e-08 ||r(i)||/||b|| 1.240037118995e-02
 52 KSP unpreconditioned resid norm 1.200360805809e-08 true resid norm 1.200360805809e-08 ||r(i)||/||b|| 1.130166101033e-02
 53 KSP unpreconditioned resid norm 1.035246990182e-08 true resid norm 1.035246990182e-08 ||r(i)||/||b|| 9.747078118834e-03
 54 KSP unpreconditioned resid norm 9.012810502968e-09 true resid norm 9.012810502968e-09 ||r(i)||/||b|| 8.485759328525e-03
 55 KSP unpreconditioned resid norm 8.556164955549e-09 true resid norm 8.556164955549e-09 ||r(i)||/||b|| 8.055817501548e-03
 56 KSP unpreconditioned resid norm 7.776893147540e-09 true resid norm 7.776893147540e-09 ||r(i)||/||b|| 7.322115953947e-03
 57 KSP unpreconditioned resid norm 6.867595067138e-09 true resid norm 6.867595067138e-09 ||r(i)||/||b|| 6.465991810912e-03
 58 KSP unpreconditioned resid norm 6.256223035332e-09 true resid norm 6.256223035332e-09 ||r(i)||/||b|| 5.890371595621e-03
 59 KSP unpreconditioned resid norm 5.775805121780e-09 true resid norm 5.775805121780e-09 ||r(i)||/||b|| 5.438047563048e-03
 60 KSP unpreconditioned resid norm 5.028152348022e-09 true resid norm 5.028152348022e-09 ||r(i)||/||b|| 4.734116031666e-03
 61 KSP unpreconditioned resid norm 4.491271029703e-09 true resid norm 4.491271029703e-09 ||r(i)||/||b|| 4.228630461573e-03
 62 KSP unpreconditioned resid norm 4.194174911407e-09 true resid norm 4.194174911407e-09 ||r(i)||/||b|| 3.948907931462e-03
 63 KSP unpreconditioned resid norm 3.900672763613e-09 true resid norm 3.900672763613e-09 ||r(i)||/||b|| 3.672569203630e-03
 64 KSP unpreconditioned resid norm 3.725382861224e-09 true resid norm 3.725382861224e-09 ||r(i)||/||b|| 3.507529904967e-03
 65 KSP unpreconditioned resid norm 3.470705216044e-09 true resid norm 3.470705216044e-09 ||r(i)||/||b|| 3.267745300304e-03
 66 KSP unpreconditioned resid norm 3.190845546802e-09 true resid norm 3.190845546802e-09 ||r(i)||/||b|| 3.004251266100e-03
 67 KSP unpreconditioned resid norm 2.936936118052e-09 true resid norm 2.936936118052e-09 ||r(i)||/||b|| 2.765189954103e-03
 68 KSP unpreconditioned resid norm 2.807750828309e-09 true resid norm 2.807750828309e-09 ||r(i)||/||b|| 2.643559162334e-03
 69 KSP unpreconditioned resid norm 2.630235180177e-09 true resid norm 2.630235180177e-09 ||r(i)||/||b|| 2.476424275098e-03
 70 KSP unpreconditioned resid norm 2.423253188367e-09 true resid norm 2.423253188367e-09 ||r(i)||/||b|| 2.281546177166e-03
 71 KSP unpreconditioned resid norm 2.312671011482e-09 true resid norm 2.312671011482e-09 ||r(i)||/||b|| 2.177430625334e-03
 72 KSP unpreconditioned resid norm 2.135449041972e-09 true resid norm 2.135449041972e-09 ||r(i)||/||b|| 2.010572242980e-03
 73 KSP unpreconditioned resid norm 2.002324106483e-09 true resid norm 2.002324106483e-09 ||r(i)||/||b|| 1.885232188086e-03
 74 KSP unpreconditioned resid norm 1.778111616174e-09 true resid norm 1.778111616174e-09 ||r(i)||/||b|| 1.674131196827e-03
 75 KSP unpreconditioned resid norm 1.653921088947e-09 true resid norm 1.653921088947e-09 ||r(i)||/||b|| 1.557203083827e-03
 76 KSP unpreconditioned resid norm 1.536016641258e-09 true resid norm 1.536016641258e-09 ||r(i)||/||b|| 1.446193452978e-03
 77 KSP unpreconditioned resid norm 1.456376200968e-09 true resid norm 1.456376200968e-09 ||r(i)||/||b|| 1.371210226725e-03
 78 KSP unpreconditioned resid norm 1.301938916885e-09 true resid norm 1.301938916885e-09 ||r(i)||/||b|| 1.225804126858e-03
 79 KSP unpreconditioned resid norm 1.256867113940e-09 true resid norm 1.256867113940e-09 ||r(i)||/||b|| 1.183368033015e-03
 80 KSP unpreconditioned resid norm 1.084746612787e-09 true resid norm 1.084746612787e-09 ||r(i)||/||b|| 1.021312795328e-03
 81 KSP unpreconditioned resid norm 1.026849960395e-09 true resid norm 1.026849960395e-09 ||r(i)||/||b|| 9.668018236432e-04
 82 KSP unpreconditioned resid norm 9.283375662057e-10 true resid norm 9.283375662057e-10 ||r(i)||/||b|| 8.740502376984e-04
KSP Object: 128 MPI processes
  type: cg
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-09, divergence=10000
  left preconditioning
  has attached null space
  using UNPRECONDITIONED norm type for convergence test
PC Object: 128 MPI processes
  type: mg
    MG: type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     128 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     128 MPI processes
      type: lu
        LU: out-of-place factorization
        tolerance for zero pivot 2.22045e-14
        matrix ordering: natural
        factor fill ratio given 0, needed 0
          Factored matrix follows:
            Matrix Object:             128 MPI processes
              type: mpiaij
              rows=1024, cols=1024
              package used to perform factorization: superlu_dist
              total: nonzeros=0, allocated nonzeros=0
              total number of mallocs used during MatSetValues calls =0
                SuperLU_DIST run parameters:
                  Process grid nprow 16 x npcol 8 
                  Equilibrate matrix TRUE 
                  Matrix input mode 1 
                  Replace tiny pivots TRUE 
                  Use iterative refinement FALSE 
                  Processors in row 16 col partition 8 
                  Row permutation LargeDiag 
                  Column permutation METIS_AT_PLUS_A
                  Parallel symbolic factorization FALSE 
                  Repeated factorization SamePattern_SameRowPerm
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=1024, cols=1024
        total: nonzeros=27648, allocated nonzeros=27648
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=8192, cols=8192
        total: nonzeros=221184, allocated nonzeros=221184
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 16 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=65536, cols=65536
        total: nonzeros=1769472, allocated nonzeros=1769472
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_3_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=524288, cols=524288
        total: nonzeros=14155776, allocated nonzeros=14155776
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     128 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=3
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_4_)     128 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       128 MPI processes
        type: mpiaij
        rows=4194304, cols=4194304
        total: nonzeros=29360128, allocated nonzeros=29360128
        total number of mallocs used during MatSetValues calls =0
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   128 MPI processes
    type: mpiaij
    rows=4194304, cols=4194304
    total: nonzeros=29360128, allocated nonzeros=29360128
    total number of mallocs used during MatSetValues calls =0
 
---------------------------------------- SUMMARY ----------------------------------------
                              Setup time =     0.0118 min 
                     Initialization time =     0.0001 min 
                         Processing time =    39.5579 min 
                    Post-processing time =     0.0028 min 
                   Total simulation time =    39.5726 min 
           Processing time per time step =     2.8391 sec 
              Total number of time steps =      836
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./hit on a interlagos-64idx-pgi-opt named nid20962 with 128 processors, by Unknown Wed Oct  2 12:24:10 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.407e+03      1.00001   2.407e+03
Objects:              3.145e+05      1.00000   3.145e+05
Flops:                3.135e+11      1.00000   3.135e+11  4.012e+13
Flops/sec:            1.303e+08      1.00001   1.303e+08  1.667e+10
MPI Messages:         6.225e+06      1.00000   6.225e+06  7.968e+08
MPI Message Lengths:  2.183e+10      1.00000   3.506e+03  2.794e+12
MPI Reductions:       7.943e+05      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.2536e+02  26.0%  5.4104e+12  13.5%  3.273e+07   4.1%  6.724e+02       19.2%  2.969e+05  37.4% 
 1:        MG Apply: 1.7813e+03  74.0%  3.4714e+13  86.5%  7.641e+08  95.9%  2.834e+03       80.8%  4.975e+05  62.6% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

VecTDot            62182 1.0 2.3234e+01 1.6 4.08e+09 1.0 0.0e+00 0.0e+00 6.2e+04  1  1  0  0  8   3 10  0  0 21 22451
VecNorm            64690 1.0 1.9899e+01 2.4 4.24e+09 1.0 0.0e+00 0.0e+00 6.5e+04  1  1  0  0  8   2 10  0  0 22 27271
VecCopy            33599 1.0 3.2168e+00 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              2530 1.0 4.4997e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY            62182 1.0 9.0091e+00 1.2 4.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1 10  0  0  0 57899
VecAYPX            62182 1.0 9.1499e+00 1.2 3.03e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  7  0  0  0 42373
VecScatterBegin    64694 1.0 8.3787e+00 1.2 0.00e+00 0.0 3.3e+07 1.6e+04 0.0e+00  0  0  4 19  0   1  0100100  0     0
VecScatterEnd      64694 1.0 3.8446e+01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   4  0  0  0  0     0
MatMult            63018 1.0 2.2957e+02 1.1 2.68e+10 1.0 3.2e+07 1.6e+04 0.0e+00  9  9  4 19  0  35 64 99 99  0 14968
MatMultTranspose       4 1.0 2.2471e-03 1.1 2.53e+05 1.0 1.5e+03 9.9e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0 14396
MatLUFactorSym         1 1.0 3.5787e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 1.3686e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin     853 1.0 2.7544e+00 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.7e+03  0  0  0  0  0   0  0  0  0  1     0
MatAssemblyEnd       853 1.0 3.0681e+00 1.2 0.00e+00 0.0 1.2e+04 1.1e+03 7.2e+01  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 2.1935e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView             5852 1.0 1.6725e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 5.9e+03  0  0  0  0  1   0  0  0  0  2     0
MatPtAP                4 1.0 2.2126e-01 1.0 5.11e+06 1.0 2.5e+04 6.0e+03 1.0e+02  0  0  0  0  0   0  0  0  0  0  2953
MatPtAPSymbolic        4 1.0 1.5837e-01 1.1 0.00e+00 0.0 1.5e+04 7.8e+03 6.0e+01  0  0  0  0  0   0  0  0  0  0     0
MatPtAPNumeric         4 1.0 7.1259e-02 1.1 5.11e+06 1.0 9.7e+03 3.1e+03 4.0e+01  0  0  0  0  0   0  0  0  0  0  9170
MatGetLocalMat         4 1.0 2.6774e-02 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          4 1.0 3.5020e-02 3.5 0.00e+00 0.0 1.1e+04 8.4e+03 8.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetSymTrans         8 1.0 9.7649e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               6 1.0 9.6231e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 2.2e+01  0  0  0  0  0   0  0  0  0  0     0
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
KSPSolve             836 1.0 2.1100e+03 1.0 3.13e+11 1.0 8.0e+08 3.5e+03 7.8e+05 88100100100 99 3377422433520264 19016
PCSetUp                1 1.0 4.3869e-01 1.0 5.36e+06 1.0 3.4e+04 4.6e+03 3.0e+02  0  0  0  0  0   0  0  0  0  0  1563
Warning -- total time of even greater than time of entire stage -- something is wrong with the timer
PCApply            31091 1.0 1.7960e+03 1.0 2.71e+11 1.0 7.6e+08 3.0e+03 5.0e+05 74 87 96 81 63 2856422335421168 19329
MGSetup Level 0        1 1.0 1.3880e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 1        1 1.0 2.2409e-0312.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 2        1 1.0 3.0804e-04 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 3        1 1.0 1.5497e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MGSetup Level 4        1 1.0 1.6789e-03 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0

--- Event Stage 1: MG Apply

VecScale          621820 1.0 3.4077e+00 1.3 1.76e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0 66146
VecCopy            31091 1.0 4.3931e-02 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet            435274 1.0 6.2440e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAYPX           124364 1.0 5.2623e+00 1.5 1.16e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0 28314
VecScatterBegin   994912 1.0 9.1571e+01 1.6 0.00e+00 0.0 7.6e+08 3.0e+03 0.0e+00  3  0 96 81  0   4  0100100  0     0
VecScatterEnd     994912 1.0 1.9157e+02 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  5  0  0  0  0   7  0  0  0  0     0
MatMult           124364 1.0 1.3612e+02 1.2 2.09e+10 1.0 1.1e+08 3.2e+03 0.0e+00  5  7 14 13  0   7  8 15 16  0 19694
MatMultAdd        124364 1.0 5.9485e+01 1.1 7.86e+09 1.0 4.8e+07 9.9e+02 0.0e+00  2  3  6  2  0   3  3  6  2  0 16907
MatMultTranspose  124364 1.0 5.8217e+01 1.4 7.86e+09 1.0 4.8e+07 9.9e+02 0.0e+00  2  3  6  2  0   3  3  6  2  0 17276
MatSolve           31091 1.0 5.5108e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   3  0  0  0  0     0
MatSOR            248728 1.0 1.4900e+03 1.0 2.33e+11 1.0 5.6e+08 3.2e+03 5.0e+05 61 74 70 65 63  83 86 73 80100 20050
KSPSolve          279819 1.0 1.5449e+03 1.0 2.33e+11 1.0 5.6e+08 3.2e+03 5.0e+05 64 74 70 65 63  86 86 73 80100 19337
PCApply            31091 1.0 5.5162e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   3  0  0  0  0     0
MGSmooth Level 0   31091 1.0 5.5668e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   3  0  0  0  0     0
MGSmooth Level 1   62182 1.0 2.0214e+01 1.2 8.54e+08 1.0 1.6e+08 1.9e+02 1.2e+05  1  0 20  1 16   1  0 21  1 25  5405
MGResid Level 1    31091 1.0 1.6500e+00 1.1 1.07e+08 1.0 3.2e+07 1.9e+02 0.0e+00  0  0  4  0  0   0  0  4  0  0  8336
MGInterp Level 1   62182 1.0 5.2735e+00 5.1 2.69e+07 1.0 2.4e+07 6.4e+01 0.0e+00  0  0  3  0  0   0  0  3  0  0   652
MGSmooth Level 2   62182 1.0 5.2998e+01 1.1 7.98e+09 1.0 1.6e+08 6.4e+02 1.2e+05  2  3 20  4 16   3  3 21  5 25 19271
MGResid Level 2    31091 1.0 4.2289e+00 1.2 8.60e+08 1.0 3.2e+07 6.4e+02 0.0e+00  0  0  4  1  0   0  0  4  1  0 26018
MGInterp Level 2   62182 1.0 3.7472e+00 1.6 2.15e+08 1.0 2.4e+07 2.1e+02 0.0e+00  0  0  3  0  0   0  0  3  0  0  7341
MGSmooth Level 3   62182 1.0 3.1296e+02 1.1 6.94e+10 1.0 1.6e+08 2.3e+03 1.2e+05 12 22 20 13 16  16 26 21 16 25 28390
MGResid Level 3    31091 1.0 2.4739e+01 1.1 6.88e+09 1.0 3.2e+07 2.3e+03 0.0e+00  1  2  4  3  0   1  3  4  3  0 35580
MGInterp Level 3   62182 1.0 1.3544e+01 1.2 1.72e+09 1.0 2.4e+07 7.7e+02 0.0e+00  0  1  3  1  0   1  1  3  1  0 16248
MGSmooth Level 4   62182 1.0 1.1547e+03 1.1 1.55e+11 1.0 8.0e+07 1.6e+04 1.2e+05 47 49 10 47 16  63 57 10 58 25 17197
MGResid Level 4    31091 1.0 1.1227e+02 1.2 1.43e+10 1.0 1.6e+07 1.6e+04 0.0e+00  4  5  2  9  0   6  5  2 12  0 16262
MGInterp Level 4   62182 1.0 9.7205e+01 1.2 1.38e+10 1.0 2.4e+07 2.9e+03 0.0e+00  4  4  3  2  0   5  5  3  3  0 18111
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Vector 65592          65592  17415948192     0
      Vector Scatter    19             19        22572     0
              Matrix    38             38     14004608     0
   Matrix Null Space     1              1          652     0
    Distributed Mesh     5              5       830792     0
     Bipartite Graph    10             10         8560     0
           Index Set    47             47       534480     0
   IS L to G Mapping     5              5       405756     0
       Krylov Solver     7              7         9536     0
     DMKSP interface     3              3         2088     0
      Preconditioner     7              7         7352     0
              Viewer     1              0            0     0

--- Event Stage 1: MG Apply

              Vector 248728          248728  19044605504     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 7.16209e-05
Average time for zero size MPI_Send(): 1.87568e-06
#PETSc Option Table entries:
-ksp_atol 1e-9
-ksp_monitor_true_residual
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type richardson
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_type mg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure run at: Wed Aug 28 23:25:43 2013
Configure options: --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok  --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
-----------------------------------------
Libraries compiled on Wed Aug 28 23:25:43 2013 on h2ologin3 
Machine characteristics: Linux-2.6.32.59-0.7-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2
Using PETSc arch: interlagos-64idx-pgi-opt
-----------------------------------------

Using C compiler: cc  -O3 -fastsse  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ftn  -O3 -fastsse   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/include -I/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/include -I/opt/cray/udreg/2.3.2-1.0402.7311.2.1.gem/include -I/opt/cray/ugni/5.0-1.0402.7128.7.6.gem/include -I/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7439.5.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7082.6.2.gem/include -I/opt/cray/xpmem/0.1-2.0402.44035.2.1.gem/include -I/opt/cray/rca/1.0.0-2.0402.42153.2.106.gem/include -I/opt/cray-hss-devel/7.0.0/include -I/opt/cray/krca/1.0.0-2.0402.42157.2.94.gem/include -I/opt/cray/mpt/6.0.1/gni/mpich2-pgi/121/include -I/opt/acml/5.3.0/pgi64_fma4/include -I/opt/cray/libsci/12.1.01/pgi/121/interlagos/include -I/opt/fftw/3.3.0.3/interlagos/include -I/usr/include/alps -I/opt/pgi/13.6.0/linux86-64/13.6/include -I/opt/cray/xe-sysroot/4.2.24/usr/include
-----------------------------------------

Using C linker: cc
Using Fortran linker: ftn
Using libraries: -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lpetsc -Wl,-rpath,/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -L/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib -lsuperlu_dist_3.3 -L/opt/acml/5.3.0/pgi64/lib -lacml -lpthread -lparmetis -lmetis -ldl 
-----------------------------------------

#PETSc Option Table entries:
-ksp_atol 1e-9
-ksp_monitor_true_residual
-ksp_view
-log_summary
-mg_coarse_pc_factor_mat_solver_package superlu_dist
-mg_coarse_pc_type lu
-mg_levels_ksp_max_it 3
-mg_levels_ksp_type richardson
-options_left
-pc_mg_galerkin
-pc_mg_levels 5
-pc_mg_log
-pc_type mg
#End of PETSc Option Table entries
There are no unused options.

From mirzadeh at gmail.com  Wed Oct  2 13:32:07 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 2 Oct 2013 11:32:07 -0700
Subject: [petsc-users] ISLocalToGlobalMapping + VecCreateGhost
In-Reply-To: <CAMYG4G=7buZxL7hdmNoX_rOxWq-wwoTagDhCZD1b-RA6yZNsdA@mail.gmail.com>
References: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>
	<CAMYG4G=7buZxL7hdmNoX_rOxWq-wwoTagDhCZD1b-RA6yZNsdA@mail.gmail.com>
Message-ID: <CAKsPKwkETt8Syy6O2r1ucnTg-xH69tWwT9OedXLrFgEhn8Qh=g@mail.gmail.com>

Matt,

I just discovered that my approach does not work properly with
VecGhostUpdateBegin/End -- I get wrong data after ghost update
process. Any idea why? When I change the IS, is PETSc aware that my
ghost values are not just at the end of vector?

Mohammad

On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
>>
>> Hi guys,
>>
>> I just did something by pure guessing which seems to work and I want
>> to make sure its the right thing!
>>
>> I have a specific layout for my vectors that look like this
>>
>>  --------------------------------------------------------------
>> | ghost values | local values | ghost values |
>>  --------------------------------------------------------------
>> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
>>
>> which means all nodes with index [0,m) and [m+n, N) are to be treated
>> as ghost and all intermediate ones as local. Since PETSc stores the
>> ghost values at the end of ghosted vec, so far I have been manually
>> mapping back and forth between petsc and my application numbering.
>> After profiling my code, it turned out that about 15% of run-time was
>> just doing this mapping which is absolutely ridiculous!
>>
>> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
>> using my own application original local2global relation shown above.
>> Then I create the petsc vec like this:
>>
>> // Create PetscVec
>> // We do not care about the actual global index of ghost nodes at this
>> point
>> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
>> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
>> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
>>
>>
>> // Apply mapping
>> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
>>
>> After this point I do the usual VecGetArray on the vec and set the
>> values, but this time without any extra mapping ... my very simple
>> test seems to be ok. Is this the correct way of using
>> ISLocalToGlobalMapping?
>
>
> Yes, this is the intention. And at a higher level, PETSc now tends to use a
> DM to
> organize this process. You tell the DM about your data layout (somewhat like
> giving
> the L2G mapping) and then you can DMGetLocalVector(), DMGetGlobalVector(),
> and
> DMLocalToGlobal().
>
>   Thanks,
>
>      Matt
>
>>
>> I guess I'm suspicious because VecCreateGhost is supposed to
>> internally set the mapping and it is supposed to position ghost nodes
>> at the end of the array which I don't want it to do...
>>
>> Thanks and sorry about the long email!
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener

From knepley at gmail.com  Wed Oct  2 13:36:04 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 Oct 2013 13:36:04 -0500
Subject: [petsc-users] ISLocalToGlobalMapping + VecCreateGhost
In-Reply-To: <CAKsPKwkETt8Syy6O2r1ucnTg-xH69tWwT9OedXLrFgEhn8Qh=g@mail.gmail.com>
References: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>
	<CAMYG4G=7buZxL7hdmNoX_rOxWq-wwoTagDhCZD1b-RA6yZNsdA@mail.gmail.com>
	<CAKsPKwkETt8Syy6O2r1ucnTg-xH69tWwT9OedXLrFgEhn8Qh=g@mail.gmail.com>
Message-ID: <CAMYG4GndxoG0H9nPgUo0G76XP-zbkX+pvg4-x0LiOws01va0Ug@mail.gmail.com>

On Wed, Oct 2, 2013 at 1:32 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Matt,
>
> I just discovered that my approach does not work properly with
> VecGhostUpdateBegin/End -- I get wrong data after ghost update
> process. Any idea why? When I change the IS, is PETSc aware that my
> ghost values are not just at the end of vector?
>

Then I was probably misunderstanding what you want to do.

  1) L2GM is for translating local to global indices automatically, but
knows nothing about "ghosting" in the Vec

  2) For ghosting, you have a local vector and global vector and a
VecScatter that maps between them

  3) VecGhost is a special kind of 2) since we know that the local vector
fits in side the global vec

  4) Since you break relationship 3), I would just use the DM (as I say
below).

  Matt


> Mohammad
>
> On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> > wrote:
> >>
> >> Hi guys,
> >>
> >> I just did something by pure guessing which seems to work and I want
> >> to make sure its the right thing!
> >>
> >> I have a specific layout for my vectors that look like this
> >>
> >>  --------------------------------------------------------------
> >> | ghost values | local values | ghost values |
> >>  --------------------------------------------------------------
> >> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
> >>
> >> which means all nodes with index [0,m) and [m+n, N) are to be treated
> >> as ghost and all intermediate ones as local. Since PETSc stores the
> >> ghost values at the end of ghosted vec, so far I have been manually
> >> mapping back and forth between petsc and my application numbering.
> >> After profiling my code, it turned out that about 15% of run-time was
> >> just doing this mapping which is absolutely ridiculous!
> >>
> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
> >> using my own application original local2global relation shown above.
> >> Then I create the petsc vec like this:
> >>
> >> // Create PetscVec
> >> // We do not care about the actual global index of ghost nodes at this
> >> point
> >> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
> >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
> >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
> >>
> >>
> >> // Apply mapping
> >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
> >>
> >> After this point I do the usual VecGetArray on the vec and set the
> >> values, but this time without any extra mapping ... my very simple
> >> test seems to be ok. Is this the correct way of using
> >> ISLocalToGlobalMapping?
> >
> >
> > Yes, this is the intention. And at a higher level, PETSc now tends to
> use a
> > DM to
> > organize this process. You tell the DM about your data layout (somewhat
> like
> > giving
> > the L2G mapping) and then you can DMGetLocalVector(),
> DMGetGlobalVector(),
> > and
> > DMLocalToGlobal().
> >
> >   Thanks,
> >
> >      Matt
> >
> >>
> >> I guess I'm suspicious because VecCreateGhost is supposed to
> >> internally set the mapping and it is supposed to position ghost nodes
> >> at the end of the array which I don't want it to do...
> >>
> >> Thanks and sorry about the long email!
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments
> > is infinitely more interesting than any results to which their
> experiments
> > lead.
> > -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/3b39996d/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct  2 13:39:12 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 Oct 2013 13:39:12 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524C6198.4000506@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
Message-ID: <7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>


  Something is wrong, you should be getting better convergence. Please answer my other email.


On Oct 2, 2013, at 1:10 PM, Michele Rosso <mrosso at uci.edu> wrote:

> Thank you all for your contribution.
> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
> 
> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
> 
> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
> I attached the diagnostics for this case. Any further idea?
> Thank you,
> 
> Michele
> 
> 
> On 10/01/2013 11:44 PM, Barry Smith wrote:
>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> 
>>> "Mark F. Adams" <mfadams at lbl.gov> writes:
>>>> run3.txt uses:
>>>> 
>>>> -ksp_type richardson
>>>> 
>>>> This is bad and I doubt anyone recommended it intentionally.
>>    Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>> 
>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>> Richardson is one of a few methods that can tolerate nonlinear
>>> preconditioners.
>>> 
>>>> You also have, in this file,
>>>> 
>>>> -mg_levels_ksp_type gmres
>>>> 
>>>> did you or the recommenders mean
>>>> 
>>>> -mg_levels_ksp_type richardson  ???
>>>> 
>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>> 
>>>> -ksp_type cg
>>>> -mg_levels_ksp_type richardson
>>>> -mg_levels_pc_type sor
>>>> 
>>>> is what I'd recommend.
>>> I thought that was tried in an earlier round.
>>> 
>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>> many more times even though the number of outer iterations
>>> 
>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>> 
>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>> 
> 
> <best.txt>


From mirzadeh at gmail.com  Wed Oct  2 13:42:09 2013
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Wed, 2 Oct 2013 11:42:09 -0700
Subject: [petsc-users] ISLocalToGlobalMapping + VecCreateGhost
In-Reply-To: <CAMYG4GndxoG0H9nPgUo0G76XP-zbkX+pvg4-x0LiOws01va0Ug@mail.gmail.com>
References: <CAKsPKwnfZB7M79Yn1FqWPt99R8KCvWc3a4sYH5GhSuHuLwhFTg@mail.gmail.com>
	<CAMYG4G=7buZxL7hdmNoX_rOxWq-wwoTagDhCZD1b-RA6yZNsdA@mail.gmail.com>
	<CAKsPKwkETt8Syy6O2r1ucnTg-xH69tWwT9OedXLrFgEhn8Qh=g@mail.gmail.com>
	<CAMYG4GndxoG0H9nPgUo0G76XP-zbkX+pvg4-x0LiOws01va0Ug@mail.gmail.com>
Message-ID: <CAKsPKw=iQKLsvnCLSSuGGFbtCK=1yO5tpHCumGt28k3ovE85-w@mail.gmail.com>

Ok I think I get it now. I'll look into DM (and probably come back
with more questions!)

Thanks

On Wed, Oct 2, 2013 at 11:36 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Oct 2, 2013 at 1:32 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
>>
>> Matt,
>>
>> I just discovered that my approach does not work properly with
>> VecGhostUpdateBegin/End -- I get wrong data after ghost update
>> process. Any idea why? When I change the IS, is PETSc aware that my
>> ghost values are not just at the end of vector?
>
>
> Then I was probably misunderstanding what you want to do.
>
>   1) L2GM is for translating local to global indices automatically, but
> knows nothing about "ghosting" in the Vec
>
>   2) For ghosting, you have a local vector and global vector and a
> VecScatter that maps between them
>
>   3) VecGhost is a special kind of 2) since we know that the local vector
> fits in side the global vec
>
>   4) Since you break relationship 3), I would just use the DM (as I say
> below).
>
>   Matt
>
>>
>> Mohammad
>>
>> On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
>> > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
>> > wrote:
>> >>
>> >> Hi guys,
>> >>
>> >> I just did something by pure guessing which seems to work and I want
>> >> to make sure its the right thing!
>> >>
>> >> I have a specific layout for my vectors that look like this
>> >>
>> >>  --------------------------------------------------------------
>> >> | ghost values | local values | ghost values |
>> >>  --------------------------------------------------------------
>> >> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
>> >>
>> >> which means all nodes with index [0,m) and [m+n, N) are to be treated
>> >> as ghost and all intermediate ones as local. Since PETSc stores the
>> >> ghost values at the end of ghosted vec, so far I have been manually
>> >> mapping back and forth between petsc and my application numbering.
>> >> After profiling my code, it turned out that about 15% of run-time was
>> >> just doing this mapping which is absolutely ridiculous!
>> >>
>> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
>> >> using my own application original local2global relation shown above.
>> >> Then I create the petsc vec like this:
>> >>
>> >> // Create PetscVec
>> >> // We do not care about the actual global index of ghost nodes at this
>> >> point
>> >> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
>> >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
>> >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
>> >>
>> >>
>> >> // Apply mapping
>> >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
>> >>
>> >> After this point I do the usual VecGetArray on the vec and set the
>> >> values, but this time without any extra mapping ... my very simple
>> >> test seems to be ok. Is this the correct way of using
>> >> ISLocalToGlobalMapping?
>> >
>> >
>> > Yes, this is the intention. And at a higher level, PETSc now tends to
>> > use a
>> > DM to
>> > organize this process. You tell the DM about your data layout (somewhat
>> > like
>> > giving
>> > the L2G mapping) and then you can DMGetLocalVector(),
>> > DMGetGlobalVector(),
>> > and
>> > DMLocalToGlobal().
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> >>
>> >> I guess I'm suspicious because VecCreateGhost is supposed to
>> >> internally set the mapping and it is supposed to position ghost nodes
>> >> at the end of the array which I don't want it to do...
>> >>
>> >> Thanks and sorry about the long email!
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments
>> > is infinitely more interesting than any results to which their
>> > experiments
>> > lead.
>> > -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener

From dave.mayhem23 at gmail.com  Wed Oct  2 17:05:26 2013
From: dave.mayhem23 at gmail.com (Dave May)
Date: Thu, 3 Oct 2013 00:05:26 +0200
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
Message-ID: <CAJ98EDo84sXF7pUyCxyV_6DjOAk8AiiZWn5jLHUaOtqVmW=6AA@mail.gmail.com>

And why are you fixed on 5 levels?

On Wednesday, 2 October 2013, Barry Smith wrote:

>
>   Something is wrong, you should be getting better convergence. Please
> answer my other email.
>
>
> On Oct 2, 2013, at 1:10 PM, Michele Rosso <mrosso at uci.edu <javascript:;>>
> wrote:
>
> > Thank you all for your contribution.
> > So far the fastest solution is still the initial one proposed by Jed in
> an earlier round:
> >
> > -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary
> -mg_coarse_pc_factor_mat_solver_package superlu_dist
> > -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type
> richardson  -options_left -pc_mg_galerkin
> > -pc_mg_levels 5  -pc_mg_log  -pc_type mg
> >
> > where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of
>  -mg_levels_ksp_max_it 1.
> > I attached the diagnostics for this case. Any further idea?
> > Thank you,
> >
> > Michele
> >
> >
> > On 10/01/2013 11:44 PM, Barry Smith wrote:
> >> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov<javascript:;>>
> wrote:
> >>
> >>> "Mark F. Adams" <mfadams at lbl.gov <javascript:;>> writes:
> >>>> run3.txt uses:
> >>>>
> >>>> -ksp_type richardson
> >>>>
> >>>> This is bad and I doubt anyone recommended it intentionally.
> >>    Hell this is normal multigrid without a Krylov accelerator. Under
> normal circumstances with geometric multigrid this should be fine, often
> the best choice.
> >>
> >>> I would have expected FGMRES, but Barry likes Krylov smoothers and
> >>> Richardson is one of a few methods that can tolerate nonlinear
> >>> preconditioners.
> >>>
> >>>> You also have, in this file,
> >>>>
> >>>> -mg_levels_ksp_type gmres
> >>>>
> >>>> did you or the recommenders mean
> >>>>
> >>>> -mg_levels_ksp_type richardson  ???
> >>>>
> >>>> you are using gmres here, which forces you to use fgmres in the outer
> solver.  This is a safe thing to use you if you apply your BCa
> symmetrically with a low order discretization then
> >>>>
> >>>> -ksp_type cg
> >>>> -mg_levels_ksp_type richardson
> >>>> -mg_levels_pc_type sor
> >>>>
> >>>> is what I'd recommend.
> >>> I thought that was tried in an earlier round.
> >>>
> >>> I don't understand why SOR preconditioning in the Krylov smoother is so
> >>> drastically more expensive than BJacobi/ILU and why SOR is called so
> >>> many more times even though the number of outer iterations
> >>>
> >>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0
> 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
> >>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0
> 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
> >>>
> >>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0
> 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
> >>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0
> 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
> >>
> >
> > <best.txt>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/79b926d2/attachment-0001.html>

From mrosso at uci.edu  Wed Oct  2 17:12:49 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Wed, 02 Oct 2013 15:12:49 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
Message-ID: <524C9A61.6030900@uci.edu>

Barry,

sorry for not replying to your other e-mail earlier.
The equation I am solving is:

$\nabla\cdot(\frac{1}{\rho}\nabla p)=\nabla\cdot u^*$

where $p$ is the pressure field, $\rho$ the density field and $u^*$  the 
velocity field.
Since I am using finite difference on a staggered grid, the pressure is 
defined on "cell" centers, while the velocity components on "cell" 
faces, even if
no cell is actually defined.
I am simulating a bi-phase flow, thus both density and pressure are 
discontinuos, but not the velocity (no mass trasfer is included at the 
moment).
Therefore the right-hand-side (rhs) of the above equation does not have 
jumps, while $p$ and $rho$ do.
In order to deal with such jumps, I am using a Ghost Fluid approach. 
Therefore the resulting linear system is slighly different from the 
typical Poisson system, though
simmetry and diagonal dominance of the matrix are mantained.
The boundary conditions are periodic in all the three space directions, 
therefore the system is singular. Thus I removed the nullspace of the 
matrix by using:

         call MatNullSpaceCreate( 
PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
                                & PETSC_NULL_INTEGER,nullspace,ierr)
         call KSPSetNullspace(ksp,nullspace,ierr)
         call MatNullSpaceDestroy(nullspace,ierr)

Hope this helps. Please let me know if you need any other info.
Also, I use the pressure at the previous time step as starting point for 
the solve. Could this be a reason why the convergence is so slow?
Thanks a lot,

Michele






On 10/02/2013 11:39 AM, Barry Smith wrote:
>    Something is wrong, you should be getting better convergence. Please answer my other email.
>
>
> On Oct 2, 2013, at 1:10 PM, Michele Rosso <mrosso at uci.edu> wrote:
>
>> Thank you all for your contribution.
>> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>>
>> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>>
>> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
>> I attached the diagnostics for this case. Any further idea?
>> Thank you,
>>
>> Michele
>>
>>
>> On 10/01/2013 11:44 PM, Barry Smith wrote:
>>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>
>>>> "Mark F. Adams" <mfadams at lbl.gov> writes:
>>>>> run3.txt uses:
>>>>>
>>>>> -ksp_type richardson
>>>>>
>>>>> This is bad and I doubt anyone recommended it intentionally.
>>>     Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>>>
>>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>>> Richardson is one of a few methods that can tolerate nonlinear
>>>> preconditioners.
>>>>
>>>>> You also have, in this file,
>>>>>
>>>>> -mg_levels_ksp_type gmres
>>>>>
>>>>> did you or the recommenders mean
>>>>>
>>>>> -mg_levels_ksp_type richardson  ???
>>>>>
>>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>>>
>>>>> -ksp_type cg
>>>>> -mg_levels_ksp_type richardson
>>>>> -mg_levels_pc_type sor
>>>>>
>>>>> is what I'd recommend.
>>>> I thought that was tried in an earlier round.
>>>>
>>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>>> many more times even though the number of outer iterations
>>>>
>>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>>>
>>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>> <best.txt>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/222b6ae8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-5.png
Type: image/png
Size: 1358 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/222b6ae8/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-4.png
Type: image/png
Size: 601 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/222b6ae8/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-3.png
Type: image/png
Size: 596 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/222b6ae8/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-2.png
Type: image/png
Size: 648 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/222b6ae8/attachment-0003.png>

From mrosso at uci.edu  Wed Oct  2 17:15:19 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Wed, 02 Oct 2013 15:15:19 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <CAJ98EDo84sXF7pUyCxyV_6DjOAk8AiiZWn5jLHUaOtqVmW=6AA@mail.gmail.com>
References: <52406460.3060905@uci.edu>	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>	<52420449.4010603@uci.edu>	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>	<524A1EA2.8090606@uci.edu>	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>	<524B7EB4.8040108@uci.edu>	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>	<87pproz1fq.fsf@mcs.anl.gov>	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>	<524C6198.4000506@uci.edu>	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<CAJ98EDo84sXF7pUyCxyV_6DjOAk8AiiZWn5jLHUaOtqVmW=6AA@mail.gmail.com>
Message-ID: <524C9AF7.3000807@uci.edu>

Dave,

I am using a 2D decomposition, so if I increase the number of levels, I 
have to decrease the number of processors I am using
in order to have enough grid points per processor for the multigrid to work.

Michele

On 10/02/2013 03:05 PM, Dave May wrote:
> And why are you fixed on 5 levels?
>
> On Wednesday, 2 October 2013, Barry Smith wrote:
>
>
>       Something is wrong, you should be getting better convergence.
>     Please answer my other email.
>
>
>     On Oct 2, 2013, at 1:10 PM, Michele Rosso <mrosso at uci.edu
>     <javascript:;>> wrote:
>
>     > Thank you all for your contribution.
>     > So far the fastest solution is still the initial one proposed by
>     Jed in an earlier round:
>     >
>     > -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view
>      -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>     > -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3
>     -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>     > -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>     >
>     > where I used  -mg_levels_ksp_max_it 3  as Barry suggested
>     instead of  -mg_levels_ksp_max_it 1.
>     > I attached the diagnostics for this case. Any further idea?
>     > Thank you,
>     >
>     > Michele
>     >
>     >
>     > On 10/01/2013 11:44 PM, Barry Smith wrote:
>     >> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov
>     <javascript:;>> wrote:
>     >>
>     >>> "Mark F. Adams" <mfadams at lbl.gov <javascript:;>> writes:
>     >>>> run3.txt uses:
>     >>>>
>     >>>> -ksp_type richardson
>     >>>>
>     >>>> This is bad and I doubt anyone recommended it intentionally.
>     >>    Hell this is normal multigrid without a Krylov accelerator.
>     Under normal circumstances with geometric multigrid this should be
>     fine, often the best choice.
>     >>
>     >>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>     >>> Richardson is one of a few methods that can tolerate nonlinear
>     >>> preconditioners.
>     >>>
>     >>>> You also have, in this file,
>     >>>>
>     >>>> -mg_levels_ksp_type gmres
>     >>>>
>     >>>> did you or the recommenders mean
>     >>>>
>     >>>> -mg_levels_ksp_type richardson  ???
>     >>>>
>     >>>> you are using gmres here, which forces you to use fgmres in
>     the outer solver.  This is a safe thing to use you if you apply
>     your BCa symmetrically with a low order discretization then
>     >>>>
>     >>>> -ksp_type cg
>     >>>> -mg_levels_ksp_type richardson
>     >>>> -mg_levels_pc_type sor
>     >>>>
>     >>>> is what I'd recommend.
>     >>> I thought that was tried in an earlier round.
>     >>>
>     >>> I don't understand why SOR preconditioning in the Krylov
>     smoother is so
>     >>> drastically more expensive than BJacobi/ILU and why SOR is
>     called so
>     >>> many more times even though the number of outer iterations
>     >>>
>     >>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09
>     1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>     >>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09
>     1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>     >>>
>     >>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10
>     1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>     >>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10
>     1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>     >>
>     >
>     > <best.txt>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/f2a0a67f/attachment.html>

From knepley at gmail.com  Wed Oct  2 17:25:02 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 Oct 2013 17:25:02 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524C9A61.6030900@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
Message-ID: <CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>

On Wed, Oct 2, 2013 at 5:12 PM, Michele Rosso <mrosso at uci.edu> wrote:

>  Barry,
>
> sorry for not replying to your other e-mail earlier.
> The equation I am solving is:
>
> [image: $\nabla\cdot(\frac{1}{\rho}\nabla p)=\nabla\cdot u^*$]
>
> where [image: $p$] is the pressure field, [image: $\rho$] the density
> field and [image: $u^*$]  the velocity field.
> Since I am using finite difference on a staggered grid, the pressure is
> defined on "cell" centers, while the velocity components on "cell" faces,
> even if
> no cell is actually defined.
>

If you just prescribe a constant rho, do you get 3-5 iterates? What about a
simple step through the domain?
Starting on the hardest problem does not sound like that way to understand
what is going on.

  Matt


> I am simulating a bi-phase flow, thus both density and pressure are
> discontinuos, but not the velocity (no mass trasfer is included at the
> moment).
> Therefore the right-hand-side (rhs) of the above equation does not have
> jumps, while $p$ and $rho$ do.
> In order to deal with such jumps, I am using a Ghost Fluid approach.
> Therefore the resulting linear system is slighly different from the typical
> Poisson system, though
> simmetry and diagonal dominance of the matrix are mantained.
> The boundary conditions are periodic in all the three space directions,
> therefore the system is singular. Thus I removed the nullspace of the
> matrix by using:
>
>         call MatNullSpaceCreate(
> PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
>                                & PETSC_NULL_INTEGER,nullspace,ierr)
>         call KSPSetNullspace(ksp,nullspace,ierr)
>         call MatNullSpaceDestroy(nullspace,ierr)
>
> Hope this helps. Please let me know if you need any other info.
> Also, I use the pressure at the previous time step as starting point for
> the solve. Could this be a reason why the convergence is so slow?
> Thanks a lot,
>
> Michele
>
>
>
>
>
>
>
> On 10/02/2013 11:39 AM, Barry Smith wrote:
>
>   Something is wrong, you should be getting better convergence. Please answer my other email.
>
>
> On Oct 2, 2013, at 1:10 PM, Michele Rosso <mrosso at uci.edu> <mrosso at uci.edu> wrote:
>
>
>  Thank you all for your contribution.
> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>
> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>
> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
> I attached the diagnostics for this case. Any further idea?
> Thank you,
>
> Michele
>
>
> On 10/01/2013 11:44 PM, Barry Smith wrote:
>
>  On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov> <jedbrown at mcs.anl.gov> wrote:
>
>
>  "Mark F. Adams" <mfadams at lbl.gov> <mfadams at lbl.gov> writes:
>
>  run3.txt uses:
>
> -ksp_type richardson
>
> This is bad and I doubt anyone recommended it intentionally.
>
>     Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>
>
>  I would have expected FGMRES, but Barry likes Krylov smoothers and
> Richardson is one of a few methods that can tolerate nonlinear
> preconditioners.
>
>
>  You also have, in this file,
>
> -mg_levels_ksp_type gmres
>
> did you or the recommenders mean
>
> -mg_levels_ksp_type richardson  ???
>
> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>
> -ksp_type cg
> -mg_levels_ksp_type richardson
> -mg_levels_pc_type sor
>
> is what I'd recommend.
>
>  I thought that was tried in an earlier round.
>
> I don't understand why SOR preconditioning in the Krylov smoother is so
> drastically more expensive than BJacobi/ILU and why SOR is called so
> many more times even though the number of outer iterations
>
> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>
> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>
>   <best.txt>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/210d1970/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-4.png
Type: image/png
Size: 601 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/210d1970/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-5.png
Type: image/png
Size: 1358 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/210d1970/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-3.png
Type: image/png
Size: 596 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/210d1970/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tblatex-2.png
Type: image/png
Size: 648 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/210d1970/attachment-0007.png>

From bsmith at mcs.anl.gov  Wed Oct  2 17:57:08 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 Oct 2013 17:57:08 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
	<CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
Message-ID: <587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>


   Ok, so piecewise constant pressures. The default DMDA multigrid is using piecewise linear interpolation. Isn't going to work well at all. You need to first add   ierr = DMDASetInterpolationType(da, DMDA_Q0);CHKERRQ(ierr); after you create the DM.  You should be seeing much better convergence. see ksp/ksp/examples/tutorials/ex34.c


   Barry



On Oct 2, 2013, at 5:25 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 2, 2013 at 5:12 PM, Michele Rosso <mrosso at uci.edu> wrote:
> Barry,
> 
> sorry for not replying to your other e-mail earlier.
> The equation I am solving is:
> 
> <tblatex-5.png>
> 
> where <tblatex-4.png> is the pressure field, <tblatex-3.png> the density field and <tblatex-2.png>  the velocity field. 
> Since I am using finite difference on a staggered grid, the pressure is defined on "cell" centers, while the velocity components on "cell" faces, even if
> no cell is actually defined.
> 
> If you just prescribe a constant rho, do you get 3-5 iterates? What about a simple step through the domain?
> Starting on the hardest problem does not sound like that way to understand what is going on.
> 
>   Matt
>  
> I am simulating a bi-phase flow, thus both density and pressure are discontinuos, but not the velocity (no mass trasfer is included at the moment).
> Therefore the right-hand-side (rhs) of the above equation does not have jumps, while $p$ and $rho$ do.
> In order to deal with such jumps, I am using a Ghost Fluid approach. Therefore the resulting linear system is slighly different from the typical Poisson system, though
> simmetry and diagonal dominance of the matrix are mantained.
> The boundary conditions are periodic in all the three space directions, therefore the system is singular. Thus I removed the nullspace of the matrix by using:
> 
>         call MatNullSpaceCreate( PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
>                                & PETSC_NULL_INTEGER,nullspace,ierr)
>         call KSPSetNullspace(ksp,nullspace,ierr)
>         call MatNullSpaceDestroy(nullspace,ierr)  
> 
> Hope this helps. Please let me know if you need any other info.
> Also, I use the pressure at the previous time step as starting point for the solve. Could this be a reason why the convergence is so slow?
> Thanks a lot,
> 
> Michele
> 
> 
> 
> 
> 
> 
> 
> On 10/02/2013 11:39 AM, Barry Smith wrote:
>>   Something is wrong, you should be getting better convergence. Please answer my other email.
>> 
>> 
>> On Oct 2, 2013, at 1:10 PM, Michele Rosso 
>> <mrosso at uci.edu>
>>  wrote:
>> 
>> 
>>> Thank you all for your contribution.
>>> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>>> 
>>> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>>> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>>> 
>>> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
>>> I attached the diagnostics for this case. Any further idea?
>>> Thank you,
>>> 
>>> Michele
>>> 
>>> 
>>> On 10/01/2013 11:44 PM, Barry Smith wrote:
>>> 
>>>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov>
>>>>  wrote:
>>>> 
>>>> 
>>>>> "Mark F. Adams" <mfadams at lbl.gov>
>>>>>  writes:
>>>>> 
>>>>>> run3.txt uses:
>>>>>> 
>>>>>> -ksp_type richardson
>>>>>> 
>>>>>> This is bad and I doubt anyone recommended it intentionally.
>>>>>> 
>>>>    Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>>>> 
>>>> 
>>>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>>>> Richardson is one of a few methods that can tolerate nonlinear
>>>>> preconditioners.
>>>>> 
>>>>> 
>>>>>> You also have, in this file,
>>>>>> 
>>>>>> -mg_levels_ksp_type gmres
>>>>>> 
>>>>>> did you or the recommenders mean
>>>>>> 
>>>>>> -mg_levels_ksp_type richardson  ???
>>>>>> 
>>>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>>>> 
>>>>>> -ksp_type cg
>>>>>> -mg_levels_ksp_type richardson
>>>>>> -mg_levels_pc_type sor
>>>>>> 
>>>>>> is what I'd recommend.
>>>>>> 
>>>>> I thought that was tried in an earlier round.
>>>>> 
>>>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>>>> many more times even though the number of outer iterations
>>>>> 
>>>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>>>> 
>>>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>>>>> 
>>> <best.txt>
>>> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From mrosso at uci.edu  Wed Oct  2 18:33:53 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Wed, 02 Oct 2013 16:33:53 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
	<CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
	<587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>
Message-ID: <524CAD61.2020004@uci.edu>

Barry,

thank you, I tried but it seems the periodic boundaries are problematic:

[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Cannot handle periodic grid in x!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./hit on a interlagos-64idx-pgi-opt named nid25146 by 
Unknown Wed Oct  2 18:22:54 2013
[0]PETSC ERROR: Libraries linked from 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib
[0]PETSC ERROR: Configure run at Wed Aug 28 23:25:43 2013
[0]PETSC ERROR: Configure options --known-level1-dcache-size=16384 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 
" --known-mpi-shared="0 " --known-memcmp-ok 
--with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" 
--COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 
-fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 
" --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " 
--with-shared-libraries=0 --with-dynamic-loading=0 
--with-mpi-compilers="1 " --known-mpi-shared-libraries=0 
--with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " 
--download-superlu_dist="1 " --download-metis="1 " 
--download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn 
PETSC_ARCH=interlagos-64idx-pgi-opt
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: DMCreateInterpolation_DA_3D_Q0() line 597 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
[0]PETSC ERROR: DMCreateInterpolation_DA() line 1015 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
[0]PETSC ERROR: DMCreateInterpolation() line 801 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/interface/dm.c
[0]PETSC ERROR: PCSetUp_MG() line 602 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: PCSetUp() line 890 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 399 in 
/mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c

On a side note I took a look at  ex34.c  and I noticed that the 
nullspace is removed not only from the matrix  but from the rhs as well.
Why is this done? Should I do the same?
Thank you.

Michele


On 10/02/2013 03:57 PM, Barry Smith wrote:
>     Ok, so piecewise constant pressures. The default DMDA multigrid is using piecewise linear interpolation. Isn't going to work well at all. You need to first add   ierr = DMDASetInterpolationType(da, DMDA_Q0);CHKERRQ(ierr); after you create the DM.  You should be seeing much better convergence. see ksp/ksp/examples/tutorials/ex34.c
>
>
>     Barry
>
>
>
> On Oct 2, 2013, at 5:25 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Oct 2, 2013 at 5:12 PM, Michele Rosso <mrosso at uci.edu> wrote:
>> Barry,
>>
>> sorry for not replying to your other e-mail earlier.
>> The equation I am solving is:
>>
>> <tblatex-5.png>
>>
>> where <tblatex-4.png> is the pressure field, <tblatex-3.png> the density field and <tblatex-2.png>  the velocity field.
>> Since I am using finite difference on a staggered grid, the pressure is defined on "cell" centers, while the velocity components on "cell" faces, even if
>> no cell is actually defined.
>>
>> If you just prescribe a constant rho, do you get 3-5 iterates? What about a simple step through the domain?
>> Starting on the hardest problem does not sound like that way to understand what is going on.
>>
>>    Matt
>>   
>> I am simulating a bi-phase flow, thus both density and pressure are discontinuos, but not the velocity (no mass trasfer is included at the moment).
>> Therefore the right-hand-side (rhs) of the above equation does not have jumps, while $p$ and $rho$ do.
>> In order to deal with such jumps, I am using a Ghost Fluid approach. Therefore the resulting linear system is slighly different from the typical Poisson system, though
>> simmetry and diagonal dominance of the matrix are mantained.
>> The boundary conditions are periodic in all the three space directions, therefore the system is singular. Thus I removed the nullspace of the matrix by using:
>>
>>          call MatNullSpaceCreate( PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
>>                                 & PETSC_NULL_INTEGER,nullspace,ierr)
>>          call KSPSetNullspace(ksp,nullspace,ierr)
>>          call MatNullSpaceDestroy(nullspace,ierr)
>>
>> Hope this helps. Please let me know if you need any other info.
>> Also, I use the pressure at the previous time step as starting point for the solve. Could this be a reason why the convergence is so slow?
>> Thanks a lot,
>>
>> Michele
>>
>>
>>
>>
>>
>>
>>
>> On 10/02/2013 11:39 AM, Barry Smith wrote:
>>>    Something is wrong, you should be getting better convergence. Please answer my other email.
>>>
>>>
>>> On Oct 2, 2013, at 1:10 PM, Michele Rosso
>>> <mrosso at uci.edu>
>>>   wrote:
>>>
>>>
>>>> Thank you all for your contribution.
>>>> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>>>>
>>>> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>>>> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>>>>
>>>> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
>>>> I attached the diagnostics for this case. Any further idea?
>>>> Thank you,
>>>>
>>>> Michele
>>>>
>>>>
>>>> On 10/01/2013 11:44 PM, Barry Smith wrote:
>>>>
>>>>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov>
>>>>>   wrote:
>>>>>
>>>>>
>>>>>> "Mark F. Adams" <mfadams at lbl.gov>
>>>>>>   writes:
>>>>>>
>>>>>>> run3.txt uses:
>>>>>>>
>>>>>>> -ksp_type richardson
>>>>>>>
>>>>>>> This is bad and I doubt anyone recommended it intentionally.
>>>>>>>
>>>>>     Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>>>>>
>>>>>
>>>>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>>>>> Richardson is one of a few methods that can tolerate nonlinear
>>>>>> preconditioners.
>>>>>>
>>>>>>
>>>>>>> You also have, in this file,
>>>>>>>
>>>>>>> -mg_levels_ksp_type gmres
>>>>>>>
>>>>>>> did you or the recommenders mean
>>>>>>>
>>>>>>> -mg_levels_ksp_type richardson  ???
>>>>>>>
>>>>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>>>>>
>>>>>>> -ksp_type cg
>>>>>>> -mg_levels_ksp_type richardson
>>>>>>> -mg_levels_pc_type sor
>>>>>>>
>>>>>>> is what I'd recommend.
>>>>>>>
>>>>>> I thought that was tried in an earlier round.
>>>>>>
>>>>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>>>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>>>>> many more times even though the number of outer iterations
>>>>>>
>>>>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>>>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>>>>>
>>>>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>>>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>>>>>>
>>>> <best.txt>
>>>>
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>


From bsmith at mcs.anl.gov  Wed Oct  2 20:22:19 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 Oct 2013 20:22:19 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524CAD61.2020004@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
	<CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
	<587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>
	<524CAD61.2020004@uci.edu>
Message-ID: <2D78269A-E3BA-4773-AC05-A6E560469DD0@mcs.anl.gov>


   You will need to fix DMCreateInterpolation_DA_3D_Q0() to support periodic boundary conditions. It might be really easy, I'm not sure why it has that condition currently.

   Barry

On Oct 2, 2013, at 6:33 PM, Michele Rosso <mrosso at uci.edu> wrote:

> Barry,
> 
> thank you, I tried but it seems the periodic boundaries are problematic:
> 
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Cannot handle periodic grid in x!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./hit on a interlagos-64idx-pgi-opt named nid25146 by Unknown Wed Oct  2 18:22:54 2013
> [0]PETSC ERROR: Libraries linked from /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib
> [0]PETSC ERROR: Configure run at Wed Aug 28 23:25:43 2013
> [0]PETSC ERROR: Configure options --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: DMCreateInterpolation_DA_3D_Q0() line 597 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
> [0]PETSC ERROR: DMCreateInterpolation_DA() line 1015 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
> [0]PETSC ERROR: DMCreateInterpolation() line 801 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/interface/dm.c
> [0]PETSC ERROR: PCSetUp_MG() line 602 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: PCSetUp() line 890 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 399 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
> 
> On a side note I took a look at  ex34.c  and I noticed that the nullspace is removed not only from the matrix  but from the rhs as well.
> Why is this done? Should I do the same?
> Thank you.
> 
> Michele
> 
> 
> On 10/02/2013 03:57 PM, Barry Smith wrote:
>>    Ok, so piecewise constant pressures. The default DMDA multigrid is using piecewise linear interpolation. Isn't going to work well at all. You need to first add   ierr = DMDASetInterpolationType(da, DMDA_Q0);CHKERRQ(ierr); after you create the DM.  You should be seeing much better convergence. see ksp/ksp/examples/tutorials/ex34.c
>> 
>> 
>>    Barry
>> 
>> 
>> 
>> On Oct 2, 2013, at 5:25 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>>> On Wed, Oct 2, 2013 at 5:12 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>> Barry,
>>> 
>>> sorry for not replying to your other e-mail earlier.
>>> The equation I am solving is:
>>> 
>>> <tblatex-5.png>
>>> 
>>> where <tblatex-4.png> is the pressure field, <tblatex-3.png> the density field and <tblatex-2.png>  the velocity field.
>>> Since I am using finite difference on a staggered grid, the pressure is defined on "cell" centers, while the velocity components on "cell" faces, even if
>>> no cell is actually defined.
>>> 
>>> If you just prescribe a constant rho, do you get 3-5 iterates? What about a simple step through the domain?
>>> Starting on the hardest problem does not sound like that way to understand what is going on.
>>> 
>>>   Matt
>>>  I am simulating a bi-phase flow, thus both density and pressure are discontinuos, but not the velocity (no mass trasfer is included at the moment).
>>> Therefore the right-hand-side (rhs) of the above equation does not have jumps, while $p$ and $rho$ do.
>>> In order to deal with such jumps, I am using a Ghost Fluid approach. Therefore the resulting linear system is slighly different from the typical Poisson system, though
>>> simmetry and diagonal dominance of the matrix are mantained.
>>> The boundary conditions are periodic in all the three space directions, therefore the system is singular. Thus I removed the nullspace of the matrix by using:
>>> 
>>>         call MatNullSpaceCreate( PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
>>>                                & PETSC_NULL_INTEGER,nullspace,ierr)
>>>         call KSPSetNullspace(ksp,nullspace,ierr)
>>>         call MatNullSpaceDestroy(nullspace,ierr)
>>> 
>>> Hope this helps. Please let me know if you need any other info.
>>> Also, I use the pressure at the previous time step as starting point for the solve. Could this be a reason why the convergence is so slow?
>>> Thanks a lot,
>>> 
>>> Michele
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 10/02/2013 11:39 AM, Barry Smith wrote:
>>>>   Something is wrong, you should be getting better convergence. Please answer my other email.
>>>> 
>>>> 
>>>> On Oct 2, 2013, at 1:10 PM, Michele Rosso
>>>> <mrosso at uci.edu>
>>>>  wrote:
>>>> 
>>>> 
>>>>> Thank you all for your contribution.
>>>>> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>>>>> 
>>>>> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>>> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>>>>> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>>>>> 
>>>>> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
>>>>> I attached the diagnostics for this case. Any further idea?
>>>>> Thank you,
>>>>> 
>>>>> Michele
>>>>> 
>>>>> 
>>>>> On 10/01/2013 11:44 PM, Barry Smith wrote:
>>>>> 
>>>>>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov>
>>>>>>  wrote:
>>>>>> 
>>>>>> 
>>>>>>> "Mark F. Adams" <mfadams at lbl.gov>
>>>>>>>  writes:
>>>>>>> 
>>>>>>>> run3.txt uses:
>>>>>>>> 
>>>>>>>> -ksp_type richardson
>>>>>>>> 
>>>>>>>> This is bad and I doubt anyone recommended it intentionally.
>>>>>>>> 
>>>>>>    Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>>>>>> 
>>>>>> 
>>>>>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>>>>>> Richardson is one of a few methods that can tolerate nonlinear
>>>>>>> preconditioners.
>>>>>>> 
>>>>>>> 
>>>>>>>> You also have, in this file,
>>>>>>>> 
>>>>>>>> -mg_levels_ksp_type gmres
>>>>>>>> 
>>>>>>>> did you or the recommenders mean
>>>>>>>> 
>>>>>>>> -mg_levels_ksp_type richardson  ???
>>>>>>>> 
>>>>>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>>>>>> 
>>>>>>>> -ksp_type cg
>>>>>>>> -mg_levels_ksp_type richardson
>>>>>>>> -mg_levels_pc_type sor
>>>>>>>> 
>>>>>>>> is what I'd recommend.
>>>>>>>> 
>>>>>>> I thought that was tried in an earlier round.
>>>>>>> 
>>>>>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>>>>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>>>>>> many more times even though the number of outer iterations
>>>>>>> 
>>>>>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>>>>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>>>>>> 
>>>>>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>>>>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>>>>>>> 
>>>>> <best.txt>
>>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
> 


From olivier.bonnefon at avignon.inra.fr  Thu Oct  3 04:48:13 2013
From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon)
Date: Thu, 03 Oct 2013 11:48:13 +0200
Subject: [petsc-users] ex12 with Neumann BC
In-Reply-To: <CAMYG4GmUQYyKLTupFzAd1JsstFykWGQBOq=KE+QAq0EBwUCmYQ@mail.gmail.com>
References: <524430F2.2020507@avignon.inra.fr>
	<CAMYG4GmUQYyKLTupFzAd1JsstFykWGQBOq=KE+QAq0EBwUCmYQ@mail.gmail.com>
Message-ID: <524D3D5D.5090301@avignon.inra.fr>

Hello,

Thank you for your answer, I'm now able to run the ex12.c with Neumann 
BC (options -bc_type neumann   -interpolate 1).

I have adapted the ex12.c for the 2D-system:

-\Delta u +u =0

It consists in adapting the fem->f0Funcs[0] function and adding the 
jacobian function fem->g0Funcs[0].
My implementation works for Dirichlet BC.
With Neumann BC(with options -bc_type neumann   -interpolate 1), the 
line search failed. I think my jacobian functions are corrects, because 
the option "-snes_mf_operator" leads to the same behavior.
Do you know what I have missed ?
In Neumann case, Where is added the 1d-integral along \delta \Omega ?


Thanks,
Olivier Bonnefon







On 09/26/2013 06:54 PM, Matthew Knepley wrote:
> On Thu, Sep 26, 2013 at 6:04 AM, Olivier Bonnefon 
> <olivier.bonnefon at avignon.inra.fr 
> <mailto:olivier.bonnefon at avignon.inra.fr>> wrote:
>
>     Hello,
>
>     I have implemented my own system from ex12.c. It works with
>     Dirichlet BC, but failed with Neumann one.
>
>     So, I'm came back to the example
>     /src/snes/example/tutorial/ex12.c, and I tried with Neumann BC:
>
>     ./ex12 -bc_type NEUMANN
>
>
> Here is the full list of tests I run (just checked that it passes in 
> 'next'):
>
> https://bitbucket.org/petsc/petsc/src/f34a81fe8510aa025c9247a5b14f0fe30e3c0bed/config/builder.py?at=master#cl-175
>
> Make sure you use an interpolated mesh with Neumann conditions since 
> you need faces.
>
>    Matt
>
>     This leads to the following crach:
>
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------------
>     [0]PETSC ERROR: No support for this operation for this object type!
>     [0]PETSC ERROR: Unsupported number of vertices 0 in cell 8 for
>     element geometry computation!
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: ./ex12 on a arch-linux2-c-debug named pcbiom38 by
>     olivierb Thu Sep 26 14:53:32 2013
>     [0]PETSC ERROR: Libraries linked from
>     /home/olivierb/SOFT/petsc-3.4.2/arch-linux2-c-debug/lib
>     [0]PETSC ERROR: Configure run at Thu Sep 26 14:44:42 2013
>     [0]PETSC ERROR: Configure options --with-debugging=1
>     --download-fiat --download-scientificpython --download-generator
>     --download-triangle --download-ctetgen --download-chaco
>     --download-netcdf --download-hdf5
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: DMPlexComputeCellGeometry() line 732 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexgeometry.c
>     [0]PETSC ERROR: DMPlexComputeResidualFEM() line 558 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexfem.c
>     [0]PETSC ERROR: SNESComputeFunction_DMLocal() line 75 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/snes/utils/dmlocalsnes.c
>     [0]PETSC ERROR: SNESComputeFunction() line 1988 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>     [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/snes/impls/ls/ls.c
>     [0]PETSC ERROR: SNESSolve() line 3636 in
>     /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>     [0]PETSC ERROR: main() line 582 in
>     "unknowndirectory/"/home/olivierb/solvers/trunk/SandBox/PETSC/LANDSCAPE/REF/ex12.c
>     --------------------------------------------------------------------------
>     MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>     with errorcode 56.
>
>     NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>     You may or may not see output from other processes, depending on
>     exactly when Open MPI kills them.
>     --------------------------------------------------------------------------
>
>     With Gbd, I saw that DMPlexGetConeSize is 0 for the last point.
>
>     Do I have forget a step to use Neumann BC ?
>
>     Thanks
>     Olivier Bonnefon
>
>     -- 
>     Olivier Bonnefon
>     INRA PACA-Avignon, Unit? BioSP
>     Tel: +33 (0)4 32 72 21 58 <tel:%2B33%20%280%294%2032%2072%2021%2058>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
Olivier Bonnefon
INRA PACA-Avignon, Unit? BioSP
Tel: +33 (0)4 32 72 21 58

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/9750a2e5/attachment-0001.html>

From mary.sweat78 at yahoo.it  Thu Oct  3 08:04:39 2013
From: mary.sweat78 at yahoo.it (mary sweat)
Date: Thu, 3 Oct 2013 14:04:39 +0100 (BST)
Subject: [petsc-users] huge sparse structured block matrix
Message-ID: <1380805479.90659.YahooMailNeo@web171806.mail.ir2.yahoo.com>

Hi all
I am very new to petsc and to c programming. I need to build a huge sparse matrix with a block structure.
The problem is that my matrx may be huge (200000*200000) so i Need a way to define it through the kronecker product or memorizing just the overdiagonal, diagonal and subdiagonal block.
Is it possible in petsc? any suggestions?
thanks a million in advance
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/98b930da/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Schermata 10-2456569 alle 15.04.06.png
Type: image/png
Size: 34505 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/98b930da/attachment-0001.png>

From knepley at gmail.com  Thu Oct  3 08:21:31 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Oct 2013 08:21:31 -0500
Subject: [petsc-users] huge sparse structured block matrix
In-Reply-To: <1380805479.90659.YahooMailNeo@web171806.mail.ir2.yahoo.com>
References: <1380805479.90659.YahooMailNeo@web171806.mail.ir2.yahoo.com>
Message-ID: <CAMYG4GnqVw9OXRzvAn-NByoVm1gMbvCjuqHEhx4j11zrqWUGCg@mail.gmail.com>

On Thu, Oct 3, 2013 at 8:04 AM, mary sweat <mary.sweat78 at yahoo.it> wrote:

> Hi all
> I am very new to petsc and to c programming. I need to build a huge sparse
> matrix with a block structure.
> The problem is that my matrx may be huge (200000*200000) so i Need a way
> to define it through the kronecker product or memorizing just the
> overdiagonal, diagonal and subdiagonal block.
> Is it possible in petsc? any suggestions?
> thanks a million in advance
>

This is the 2D Laplacian with a 5-point FD stencil. There are many PETSc
examples which
form and solve this. For example, here is an example:

  1,640,961 unknowns (on the fine level)

  8,199,681 nonzeros

  ./ex5 -da_grid_x 21 -da_grid_y 21 -da_refine 6 -ksp_rtol 1.0e-9 -pc_type
mg -pc_mg_levels 4 -snes_monitor -snes_view

This takes 375s on my underpowered Mac Air laptop.

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/047299db/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct  3 08:40:04 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 3 Oct 2013 08:40:04 -0500
Subject: [petsc-users] huge sparse structured block matrix
In-Reply-To: <CAMYG4GnqVw9OXRzvAn-NByoVm1gMbvCjuqHEhx4j11zrqWUGCg@mail.gmail.com>
References: <1380805479.90659.YahooMailNeo@web171806.mail.ir2.yahoo.com>
	<CAMYG4GnqVw9OXRzvAn-NByoVm1gMbvCjuqHEhx4j11zrqWUGCg@mail.gmail.com>
Message-ID: <CAM9tzSn=5M-b5E5NwGp2fudapdxkHZNGXnys4ePZqAx8WT-TTw@mail.gmail.com>

... in debug mode. (Optimized is much faster.)

And note that this example is the nonlinear Bratu problem, which while very
similar to a Laplacian, is not the same and performs multiple linear solves.
On Oct 3, 2013 8:21 AM, "Matthew Knepley" <knepley at gmail.com> wrote:

> On Thu, Oct 3, 2013 at 8:04 AM, mary sweat <mary.sweat78 at yahoo.it> wrote:
>
>>  Hi all
>> I am very new to petsc and to c programming. I need to build a huge
>> sparse matrix with a block structure.
>> The problem is that my matrx may be huge (200000*200000) so i Need a way
>> to define it through the kronecker product or memorizing just the
>> overdiagonal, diagonal and subdiagonal block.
>> Is it possible in petsc? any suggestions?
>> thanks a million in advance
>>
>
> This is the 2D Laplacian with a 5-point FD stencil. There are many PETSc
> examples which
> form and solve this. For example, here is an example:
>
>   1,640,961 unknowns (on the fine level)
>
>   8,199,681 nonzeros
>
>   ./ex5 -da_grid_x 21 -da_grid_y 21 -da_refine 6 -ksp_rtol 1.0e-9 -pc_type
> mg -pc_mg_levels 4 -snes_monitor -snes_view
>
> This takes 375s on my underpowered Mac Air laptop.
>
>   Thanks,
>
>      Matt
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/612dbb4f/attachment.html>

From wumengda at gmail.com  Thu Oct  3 08:54:48 2013
From: wumengda at gmail.com (Mengda Wu)
Date: Thu, 3 Oct 2013 09:54:48 -0400
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <alpine.LFD.2.03.1310020004130.1897@mcs.anl.gov>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
	<alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
	<CALmcJSCsvmyUdx_RjdSLqbXt_5D7oosaJk02M=6a_2MOOcsdHw@mail.gmail.com>
	<alpine.LFD.2.03.1310020004130.1897@mcs.anl.gov>
Message-ID: <CALmcJSB71q8A8i=vyKc5AghVsxBAS9OuNQx0ByHms==DOctUHg@mail.gmail.com>

Hi Satis,

  I've tried to use mingw distributed with cygwin. And it worked well for
static libraries.

  Another problem I am facing is I cannot build shared libraries with it
even I supplied with "--with-shared-libraries=1". Only .a libs are
generated and there are no dlls. There was a message saying "CMake setup
incomplete (status 256), falling back to legacy build" when configuring. I
did install Cygwin's cmake. I don't know if cmake will help for shared
libraries.

Thanks,
Mengda


On Wed, Oct 2, 2013 at 1:07 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 1 Oct 2013, Mengda Wu wrote:
>
> > Hi Satish,
> >
> >    Thanks a lot for your reply!
> >
> >    Is there a way to use the mingw64 which is not shipped with cygwin?
> For
> > example, use windows native paths when testing the compiler.
>
> Nope - I don't think it will work. Our build tools are primarily work with
> cygwin paths.
>
> Alternative is to use MS/Intel compilers - and we support them with
> win32fe compiler wrapper.
>
> What problem are you having with using mingw distributed with cygwin?
>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/fb31368a/attachment.html>

From rlmackie862 at gmail.com  Thu Oct  3 14:07:06 2013
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 3 Oct 2013 12:07:06 -0700
Subject: [petsc-users] VecScatter from slice of global vector to sequential
	vector + AO errors
Message-ID: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>

I am trying to create a VecScatter that will scatter all the values from a
horizontal slice of a 3D DMDA global vector (with dof=3) to a sequential
vector on every processor. So far I have been unsuccessful, most likely
because I don't completely understand how to get the appropriate IS to do
this. I believe that you need to create a sequential IS on every processor
that has the PETSc global number for the appropriate values I want to
scatter. What I most recently tried was the following code which takes the
values at level k:

  DM          :: da3
  Vec         :: daSlice, daVec
  PetscInt    :: i, j, k, ierr, nis, istart, istep
  IS          :: from, to
  AO          :: ntog
  VecScatter  :: vscat
  PetscInt, allocatable :: indices(:)

  ! Get the AO context for this DM
  call DMDAGetAO(da3,ntog,ierr)

  ! Create indices for scatter from global vector
  allocate (indices(3*nx*ny))
  istart=(k-1)*nx*ny*3
  nis=nx*ny*3
  do i=1,nis
    indices(i)=istart+i-1
  end do

  ! Convert the indices from natural ordering to PETSc ordering
  call AOApplicationToPetsc(ntog,nis,indices,ierr)

  call
ISCreateGeneral(PETSC_COMM_SELF,nis,indices,PETSC_COPY_VALUES,from,ierr)

  ! Get the index set that specifies elements to scatter to sequential
vector
  istep=1
  istart=0
  nis=3*nx*ny
  call ISCreateStride(PETSC_COMM_SELF,nis,istart,istep,to,ierr)

  ! Create the Vector Scatter to get the values to every processor
  call VecScatterCreate(daVec,from,daSlice,to,vscat,ierr)

  ! Release memory
  deallocate (indices)
  call AODestroy(ntog,ierr)
  call ISDestroy(from,ierr)
  call ISDestroy(to,ierr)


This code is throwing errors as follows

[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Invalid argument!
[2]PETSC ERROR: Wrong type of object: Parameter # 1!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Libraries linked from
/home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
[2]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
[2]PETSC ERROR: Configure options --with-scalar-type=complex
--with-debugging=1 --with-fortran=1 --download-mpich=1
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: AOApplicationToPetsc() line 261 in
/home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Invalid argument!
[1]PETSC ERROR: Wrong type of object: Parameter # 1!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Libraries linked from
/home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
[1]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
[1]PETSC ERROR: Configure options --with-scalar-type=complex
--with-debugging=1 --with-fortran=1 --download-mpich=1
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: AOApplicationToPetsc() line 261 in
/home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: -----------------------------------

etc.


First, I don't see where my error is that is causing these problems, but
second, is there a better way to create this vector scatter without having
to use an AO? I suspect there is,  but so far I have been unable to figure
out how to create the correct IS.

Thanks very much,

Randy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/30caf79c/attachment.html>

From knepley at gmail.com  Thu Oct  3 14:24:49 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 Oct 2013 14:24:49 -0500
Subject: [petsc-users] VecScatter from slice of global vector to
 sequential vector + AO errors
In-Reply-To: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>
References: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>
Message-ID: <CAMYG4GkO1iovSfUrs+ZK8fmoYL+524PCeSBeHwx4E_R9PhZfWw@mail.gmail.com>

On Thu, Oct 3, 2013 at 2:07 PM, Randall Mackie <rlmackie862 at gmail.com>wrote:

> I am trying to create a VecScatter that will scatter all the values from a
> horizontal slice of a 3D DMDA global vector (with dof=3) to a sequential
> vector on every processor. So far I have been unsuccessful, most likely
> because I don't completely understand how to get the appropriate IS to do
> this. I believe that you need to create a sequential IS on every processor
> that has the PETSc global number for the appropriate values I want to
> scatter. What I most recently tried was the following code which takes the
> values at level k:
>
>   DM          :: da3
>   Vec         :: daSlice, daVec
>   PetscInt    :: i, j, k, ierr, nis, istart, istep
>   IS          :: from, to
>   AO          :: ntog
>   VecScatter  :: vscat
>   PetscInt, allocatable :: indices(:)
>
>   ! Get the AO context for this DM
>   call DMDAGetAO(da3,ntog,ierr)
>
>   ! Create indices for scatter from global vector
>   allocate (indices(3*nx*ny))
>   istart=(k-1)*nx*ny*3
>   nis=nx*ny*3
>   do i=1,nis
>     indices(i)=istart+i-1
>   end do
>
>   ! Convert the indices from natural ordering to PETSc ordering
>   call AOApplicationToPetsc(ntog,nis,indices,ierr)
>
>   call
> ISCreateGeneral(PETSC_COMM_SELF,nis,indices,PETSC_COPY_VALUES,from,ierr)
>
>   ! Get the index set that specifies elements to scatter to sequential
> vector
>   istep=1
>   istart=0
>   nis=3*nx*ny
>   call ISCreateStride(PETSC_COMM_SELF,nis,istart,istep,to,ierr)
>
>   ! Create the Vector Scatter to get the values to every processor
>   call VecScatterCreate(daVec,from,daSlice,to,vscat,ierr)
>
>   ! Release memory
>   deallocate (indices)
>   call AODestroy(ntog,ierr)
>   call ISDestroy(from,ierr)
>   call ISDestroy(to,ierr)
>
>
> This code is throwing errors as follows
>
> [2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Invalid argument!
> [2]PETSC ERROR: Wrong type of object: Parameter # 1!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Libraries linked from
> /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [2]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [2]PETSC ERROR: Configure options --with-scalar-type=complex
> --with-debugging=1 --with-fortran=1 --download-mpich=1
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: AOApplicationToPetsc() line 261 in
> /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Invalid argument!
> [1]PETSC ERROR: Wrong type of object: Parameter # 1!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Libraries linked from
> /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [1]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [1]PETSC ERROR: Configure options --with-scalar-type=complex
> --with-debugging=1 --with-fortran=1 --download-mpich=1
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: AOApplicationToPetsc() line 261 in
> /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: -----------------------------------
>
> etc.
>
>
> First, I don't see where my error is that is causing these problems, but
> second, is there a better way to create this vector scatter without having
> to use an AO? I suspect there is,  but so far I have been unable to figure
> out how to create the correct IS.
>

I think this is the best way to do what you want. The error says that
DMDAGetAO() does not actually give you an AO.
Can you see what you do get in the debugger? I am wondering if you have
overwrite in one of your index arrays.

  Matt


> Thanks very much,
>
> Randy
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/7e2e246d/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct  3 14:31:04 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 Oct 2013 14:31:04 -0500
Subject: [petsc-users] VecScatter from slice of global vector to
	sequential vector + AO errors
In-Reply-To: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>
References: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>
Message-ID: <ACC82763-8222-416B-B870-EC0899D8BC4D@mcs.anl.gov>


  Randy,

     Take a look at DMDAGetRay() in src/dm/impls/da/dasub.c (now called DMDACreateRay()) this takes a row or column from a 2d DADM. You can use the same kind of approach to get a slice from a 3d DMDA.

   Barry




On Oct 3, 2013, at 2:07 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:

> I am trying to create a VecScatter that will scatter all the values from a horizontal slice of a 3D DMDA global vector (with dof=3) to a sequential vector on every processor. So far I have been unsuccessful, most likely because I don't completely understand how to get the appropriate IS to do this. I believe that you need to create a sequential IS on every processor that has the PETSc global number for the appropriate values I want to scatter. What I most recently tried was the following code which takes the values at level k:
> 
>   DM          :: da3
>   Vec         :: daSlice, daVec
>   PetscInt    :: i, j, k, ierr, nis, istart, istep
>   IS          :: from, to
>   AO          :: ntog
>   VecScatter  :: vscat
>   PetscInt, allocatable :: indices(:)
> 
>   ! Get the AO context for this DM
>   call DMDAGetAO(da3,ntog,ierr)
> 
>   ! Create indices for scatter from global vector
>   allocate (indices(3*nx*ny))
>   istart=(k-1)*nx*ny*3
>   nis=nx*ny*3
>   do i=1,nis
>     indices(i)=istart+i-1
>   end do
> 
>   ! Convert the indices from natural ordering to PETSc ordering
>   call AOApplicationToPetsc(ntog,nis,indices,ierr)
> 
>   call ISCreateGeneral(PETSC_COMM_SELF,nis,indices,PETSC_COPY_VALUES,from,ierr)
> 
>   ! Get the index set that specifies elements to scatter to sequential vector
>   istep=1
>   istart=0
>   nis=3*nx*ny
>   call ISCreateStride(PETSC_COMM_SELF,nis,istart,istep,to,ierr)
> 
>   ! Create the Vector Scatter to get the values to every processor
>   call VecScatterCreate(daVec,from,daSlice,to,vscat,ierr)
> 
>   ! Release memory
>   deallocate (indices)
>   call AODestroy(ntog,ierr)
>   call ISDestroy(from,ierr)
>   call ISDestroy(to,ierr)
> 
> 
> This code is throwing errors as follows
> 
> [2]PETSC ERROR: --------------------- Error Message ------------------------------------
> [2]PETSC ERROR: Invalid argument!
> [2]PETSC ERROR: Wrong type of object: Parameter # 1!
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Libraries linked from /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [2]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [2]PETSC ERROR: Configure options --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich=1
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: AOApplicationToPetsc() line 261 in /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Invalid argument!
> [1]PETSC ERROR: Wrong type of object: Parameter # 1!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Libraries linked from /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [1]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich=1
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: AOApplicationToPetsc() line 261 in /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: -----------------------------------
> 
> etc.
> 
> 
> First, I don't see where my error is that is causing these problems, but second, is there a better way to create this vector scatter without having to use an AO? I suspect there is,  but so far I have been unable to figure out how to create the correct IS.
> 
> Thanks very much,
> 
> Randy
> 


From mrosso at uci.edu  Thu Oct  3 16:41:04 2013
From: mrosso at uci.edu (Michele Rosso)
Date: Thu, 03 Oct 2013 14:41:04 -0700
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <2D78269A-E3BA-4773-AC05-A6E560469DD0@mcs.anl.gov>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
	<CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
	<587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>
	<524CAD61.2020004@uci.edu>
	<2D78269A-E3BA-4773-AC05-A6E560469DD0@mcs.anl.gov>
Message-ID: <524DE470.5080308@uci.edu>

Barry,

thank you, I hope to be able to fix this bug.
On a side note, I took a look at ex34.c and I noticed that the nullspace 
is removed not only from the matrix but from the rhs as well. Why is 
this done?

Thank you.
  Michele




On 10/02/2013 06:22 PM, Barry Smith wrote:
>     You will need to fix DMCreateInterpolation_DA_3D_Q0() to support periodic boundary conditions. It might be really easy, I'm not sure why it has that condition currently.
>
>     Barry
>
> On Oct 2, 2013, at 6:33 PM, Michele Rosso <mrosso at uci.edu> wrote:
>
>> Barry,
>>
>> thank you, I tried but it seems the periodic boundaries are problematic:
>>
>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Invalid argument!
>> [0]PETSC ERROR: Cannot handle periodic grid in x!
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./hit on a interlagos-64idx-pgi-opt named nid25146 by Unknown Wed Oct  2 18:22:54 2013
>> [0]PETSC ERROR: Libraries linked from /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/interlagos-64idx-pgi-opt/lib
>> [0]PETSC ERROR: Configure run at Wed Aug 28 23:25:43 2013
>> [0]PETSC ERROR: Configure options --known-level1-dcache-size=16384 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=4 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=0 --known-mpi-c-double-complex=0 --with-batch="1 " --known-mpi-shared="0 " --known-memcmp-ok --with-blas-lapack-lib="-L/opt/acml/5.3.0/pgi64/lib  -lacml" --COPTFLAGS="-O3 -fastsse" --FOPTFLAGS="-O3 -fastsse" --CXXOPTFLAGS="-O3 -fastsse" --with-x="0 " --with-debugging="0 " --with-clib-autodetect="0 " --with-cxxlib-autodetect="0 " --with-fortranlib-autodetect="0 " --with-shared-libraries=0 --with-dynamic-loading=0 --with-mpi-compilers="1 " --known-mpi-shared-libraries=0 --with-64-bit-indices --download-blacs="1 " --download-scalapack="1 " --download-superlu_dist="1 " --download-metis="1 " --download-parmetis="1 " --with-cc=cc --with-cxx=CC --with-fc=ftn PETSC_ARCH=interlagos-64idx-pgi-opt
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: DMCreateInterpolation_DA_3D_Q0() line 597 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
>> [0]PETSC ERROR: DMCreateInterpolation_DA() line 1015 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/impls/da/dainterp.c
>> [0]PETSC ERROR: DMCreateInterpolation() line 801 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/dm/interface/dm.c
>> [0]PETSC ERROR: PCSetUp_MG() line 602 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/impls/mg/mg.c
>> [0]PETSC ERROR: PCSetUp() line 890 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: KSPSetUp() line 278 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: KSPSolve() line 399 in /mnt/a/u/sciteam/mrosso/LIBS/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
>>
>> On a side note I took a look at  ex34.c  and I noticed that the nullspace is removed not only from the matrix  but from the rhs as well.
>> Why is this done? Should I do the same?
>> Thank you.
>>
>> Michele
>>
>>
>> On 10/02/2013 03:57 PM, Barry Smith wrote:
>>>     Ok, so piecewise constant pressures. The default DMDA multigrid is using piecewise linear interpolation. Isn't going to work well at all. You need to first add   ierr = DMDASetInterpolationType(da, DMDA_Q0);CHKERRQ(ierr); after you create the DM.  You should be seeing much better convergence. see ksp/ksp/examples/tutorials/ex34.c
>>>
>>>
>>>     Barry
>>>
>>>
>>>
>>> On Oct 2, 2013, at 5:25 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>>> On Wed, Oct 2, 2013 at 5:12 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>>> Barry,
>>>>
>>>> sorry for not replying to your other e-mail earlier.
>>>> The equation I am solving is:
>>>>
>>>> <tblatex-5.png>
>>>>
>>>> where <tblatex-4.png> is the pressure field, <tblatex-3.png> the density field and <tblatex-2.png>  the velocity field.
>>>> Since I am using finite difference on a staggered grid, the pressure is defined on "cell" centers, while the velocity components on "cell" faces, even if
>>>> no cell is actually defined.
>>>>
>>>> If you just prescribe a constant rho, do you get 3-5 iterates? What about a simple step through the domain?
>>>> Starting on the hardest problem does not sound like that way to understand what is going on.
>>>>
>>>>    Matt
>>>>   I am simulating a bi-phase flow, thus both density and pressure are discontinuos, but not the velocity (no mass trasfer is included at the moment).
>>>> Therefore the right-hand-side (rhs) of the above equation does not have jumps, while $p$ and $rho$ do.
>>>> In order to deal with such jumps, I am using a Ghost Fluid approach. Therefore the resulting linear system is slighly different from the typical Poisson system, though
>>>> simmetry and diagonal dominance of the matrix are mantained.
>>>> The boundary conditions are periodic in all the three space directions, therefore the system is singular. Thus I removed the nullspace of the matrix by using:
>>>>
>>>>          call MatNullSpaceCreate( PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,&
>>>>                                 & PETSC_NULL_INTEGER,nullspace,ierr)
>>>>          call KSPSetNullspace(ksp,nullspace,ierr)
>>>>          call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> Hope this helps. Please let me know if you need any other info.
>>>> Also, I use the pressure at the previous time step as starting point for the solve. Could this be a reason why the convergence is so slow?
>>>> Thanks a lot,
>>>>
>>>> Michele
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/02/2013 11:39 AM, Barry Smith wrote:
>>>>>    Something is wrong, you should be getting better convergence. Please answer my other email.
>>>>>
>>>>>
>>>>> On Oct 2, 2013, at 1:10 PM, Michele Rosso
>>>>> <mrosso at uci.edu>
>>>>>   wrote:
>>>>>
>>>>>
>>>>>> Thank you all for your contribution.
>>>>>> So far the fastest solution is still the initial one proposed by Jed in an earlier round:
>>>>>>
>>>>>> -ksp_atol 1e-9  -ksp_monitor_true_residual  -ksp_view  -log_summary -mg_coarse_pc_factor_mat_solver_package superlu_dist
>>>>>> -mg_coarse_pc_type lu    -mg_levels_ksp_max_it 3 -mg_levels_ksp_type richardson  -options_left -pc_mg_galerkin
>>>>>> -pc_mg_levels 5  -pc_mg_log  -pc_type mg
>>>>>>
>>>>>> where I used  -mg_levels_ksp_max_it 3  as Barry suggested instead of  -mg_levels_ksp_max_it 1.
>>>>>> I attached the diagnostics for this case. Any further idea?
>>>>>> Thank you,
>>>>>>
>>>>>> Michele
>>>>>>
>>>>>>
>>>>>> On 10/01/2013 11:44 PM, Barry Smith wrote:
>>>>>>
>>>>>>> On Oct 2, 2013, at 12:28 AM, Jed Brown <jedbrown at mcs.anl.gov>
>>>>>>>   wrote:
>>>>>>>
>>>>>>>
>>>>>>>> "Mark F. Adams" <mfadams at lbl.gov>
>>>>>>>>   writes:
>>>>>>>>
>>>>>>>>> run3.txt uses:
>>>>>>>>>
>>>>>>>>> -ksp_type richardson
>>>>>>>>>
>>>>>>>>> This is bad and I doubt anyone recommended it intentionally.
>>>>>>>>>
>>>>>>>     Hell this is normal multigrid without a Krylov accelerator. Under normal circumstances with geometric multigrid this should be fine, often the best choice.
>>>>>>>
>>>>>>>
>>>>>>>> I would have expected FGMRES, but Barry likes Krylov smoothers and
>>>>>>>> Richardson is one of a few methods that can tolerate nonlinear
>>>>>>>> preconditioners.
>>>>>>>>
>>>>>>>>
>>>>>>>>> You also have, in this file,
>>>>>>>>>
>>>>>>>>> -mg_levels_ksp_type gmres
>>>>>>>>>
>>>>>>>>> did you or the recommenders mean
>>>>>>>>>
>>>>>>>>> -mg_levels_ksp_type richardson  ???
>>>>>>>>>
>>>>>>>>> you are using gmres here, which forces you to use fgmres in the outer solver.  This is a safe thing to use you if you apply your BCa symmetrically with a low order discretization then
>>>>>>>>>
>>>>>>>>> -ksp_type cg
>>>>>>>>> -mg_levels_ksp_type richardson
>>>>>>>>> -mg_levels_pc_type sor
>>>>>>>>>
>>>>>>>>> is what I'd recommend.
>>>>>>>>>
>>>>>>>> I thought that was tried in an earlier round.
>>>>>>>>
>>>>>>>> I don't understand why SOR preconditioning in the Krylov smoother is so
>>>>>>>> drastically more expensive than BJacobi/ILU and why SOR is called so
>>>>>>>> many more times even though the number of outer iterations
>>>>>>>>
>>>>>>>> bjacobi: PCApply              322 1.0 4.1021e+01 1.0 6.44e+09 1.0 3.0e+07 1.6e+03 4.5e+04 74 86 98 88 92 28160064317351226 20106
>>>>>>>> bjacobi: KSPSolve              46 1.0 4.6268e+01 1.0 7.52e+09 1.0 3.0e+07 1.8e+03 4.8e+04 83100100 99 99 31670065158291309 20800
>>>>>>>>
>>>>>>>> sor:     PCApply             1132 1.0 1.5532e+02 1.0 2.30e+10 1.0 1.0e+08 1.6e+03 1.6e+05 69 88 99 88 93 21871774317301274 18987
>>>>>>>> sor:     KSPSolve             201 1.0 1.7101e+02 1.0 2.63e+10 1.0 1.1e+08 1.8e+03 1.7e+05 75100100 99 98 24081775248221352 19652
>>>>>>>>
>>>>>> <best.txt>
>>>>>>
>>>>
>>>>
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>



From jedbrown at mcs.anl.gov  Thu Oct  3 16:43:13 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 03 Oct 2013 16:43:13 -0500
Subject: [petsc-users] How to speed up geometric multigrid
In-Reply-To: <524DE470.5080308@uci.edu>
References: <52406460.3060905@uci.edu>
	<4499926F-40CE-462C-BFD6-8798A02126FB@mcs.anl.gov>
	<52420449.4010603@uci.edu>
	<60076751-799B-410A-9779-D697677A869D@mcs.anl.gov>
	<524A1EA2.8090606@uci.edu>
	<9079A4B7-C533-4F90-BA8E-C163761A4A99@mcs.anl.gov>
	<524B7EB4.8040108@uci.edu>
	<D21D09E2-94C5-48C2-A3C3-E19461C52638@lbl.gov>
	<87pproz1fq.fsf@mcs.anl.gov>
	<B775904E-DB54-4B68-82C5-AB8B65BA398E@mcs.anl.gov>
	<524C6198.4000506@uci.edu>
	<7229EA34-CB99-44B3-B4CE-DE5F71CF141A@mcs.anl.gov>
	<524C9A61.6030900@uci.edu>
	<CAMYG4Gm+v8kj_fjLQOTxe7frHG=v29jDh8yOZy+=+6FZU8VVDQ@mail.gmail.com>
	<587DEDB2-AA71-41BA-BBD0-7A90963EB526@mcs.anl.gov>
	<524CAD61.2020004@uci.edu>
	<2D78269A-E3BA-4773-AC05-A6E560469DD0@mcs.anl.gov>
	<524DE470.5080308@uci.edu>
Message-ID: <87mwmq58um.fsf@mcs.anl.gov>

Michele Rosso <mrosso at uci.edu> writes:

> Barry,
>
> thank you, I hope to be able to fix this bug.
> On a side note, I took a look at ex34.c and I noticed that the nullspace 
> is removed not only from the matrix but from the rhs as well. Why is 
> this done?

You need the right hand side to be consistent if you want to be able to
say that your residual is small.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/861ee48c/attachment.pgp>

From jed at 59A2.org  Thu Oct  3 17:18:27 2013
From: jed at 59A2.org (Jed Brown)
Date: Thu, 03 Oct 2013 17:18:27 -0500
Subject: [petsc-users] Strong and weak scaling,
	solver robustness [was Re: reference]
In-Reply-To: <524C3C8D.9010608@gfz-potsdam.de>
References: <524BF29C.9040307@gfz-potsdam.de> <87haczzyb2.fsf@mcs.anl.gov>
	<524C3C8D.9010608@gfz-potsdam.de>
Message-ID: <87d2nm577w.fsf@mcs.anl.gov>

Cc'ing petsc-users since this may be of interest there.

Alexander Grayver <agrayver at gfz-potsdam.de> writes:

> Jed,
>
> thanks.
>
> I actually realized that I don't quite understand the definition of 
> scalable as "if its performance is independent of resolution and 
> parallelism". Could you shortly comment on what is resolution here?
> I thought it is more conventional to define scalability in terms of 
> strong and weak graphs?

Strong scaling is well-defined in the sense that you fix the problem and
ask how fast you can solve it by adding parallelism.  It has two parts,
the implementation quality (which can include some math related to
structuring a given communication pattern) and algorithmic adaptations
to expose increasingly-fine-grained parallelism.  The methods
appropriate on a serial CPU are quite different from those appropriate
on a GPU or a supercomputer, so it would be unfair to compare the same
method in both places.  We can ask that the algorithmic changes needed
to expose fine-grained parallelism in a fixed target problem (e.g.,
moving from a workstation to a supercomputer) not affect convergence too
much.

Weak scaling is trickier to define.  Supposing we are solving a linear
PDE.  If you take a fixed geometry and coefficient field and then refine
the hell out of the mesh, you are creating much easier (smoother)
problems where the original discretization doesn't make sense.  That is,
for a rough coefficient field, you probably use a low-order compatible
discretization, but as you go to high accuracy on very smooth solutions,
you _should_ have switched to a high order method.  Similarly, multigrid
is much easier because the grid hierarchy doesn't have to homogenize
because solutions are smooth/resolved throughout the grid hierarchy.

So instead, the only fair thing to do with weak scaling is to solve
problems people could care about at the higher resolution.  This either
means looking at bigger domains filled with heterogeneity at the same
scale, or adding further fine structure.  That fine structure needs to
not have some "easy" structure like scale separation lest you have
anomalous convergence behavior.  If the target problem is nonlinear, you
would like the nonlinearity to have the same stiffness at all scales.
Nonlinearities are rarely scale-free.

Consequently, for many applications, it's not possible to do a "fair"
weak scaling.  You can solve a collection of problems of different
sizes, but you can't hope for the convergence behavior to be directly
comparable.


You can test a solver for robustness in the sense of taking a fixed
problem size and watching the convergence behavior as you manipulate the
material coefficients and parameters.  You'll almost never converge at a
rate truly independent of those parameters when using a scalable method.

> On 02.10.2013 13:50, Jed Brown wrote:
>> Alexander Grayver <agrayver at gfz-potsdam.de> writes:
>>
>>> Hi Jed,
>>>
>>> I liked your definition of scalable, robust and efficient solvers in CIG
>>> Webinar.
>>> Have you got any paper or proceedings where you talk about these issues
>>> so that I could cite it?
>> Hmm, I have a conference position paper about new research in support of
>> those ideals.
>>
>> @inproceedings{brown2013dimension,
>>    title={Vectorization, communication aggregation, and reuse in stochastic and temporal dimensions},
>>    author={Jed Brown},
>>    booktitle={Exascale Mathematics Workshop, Aug 21--22, Washington, DC},
>>    year={2013},
>>    organization={DOE Office of Advanced Scientific Computing Research}
>> }
>>
>> Bill Gropp has some relevant software papers.
>>
>> @article{gropp2005issues,
>>    title={{Issues in accurate and reliable use of parallel computing in numerical programs}},
>>    author={Gropp, W.D.},
>>    journal={Accuracy and Reliability in Scientific Computing},
>>    pages={253--263},
>>    year={2005}
>> }
>>
>> @inproceedings{gropp1999exploiting,
>>    title={{Exploiting existing software in libraries: Successes, Failures, and Reasons Why}},
>>    author={Gropp, W.D.},
>>    booktitle={Proceedings of the SIAM Workshop on Object Oriented Methods for Interoperable Scientific and Engineering Computing},
>>    pages={21--29},
>>    isbn={0898714451},
>>    year={1999},
>>    organization={Soc for Industrial \& Applied Math},
>>    publisher={SIAM}
>> }
>>
>>
>
>
> -- 
> Regards,
> Alexander
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/23915515/attachment.pgp>

From rlmackie862 at gmail.com  Thu Oct  3 18:09:08 2013
From: rlmackie862 at gmail.com (Randall Mackie)
Date: Thu, 3 Oct 2013 16:09:08 -0700
Subject: [petsc-users] VecScatter from slice of global vector to
	sequential vector + AO errors
In-Reply-To: <CAMYG4GkO1iovSfUrs+ZK8fmoYL+524PCeSBeHwx4E_R9PhZfWw@mail.gmail.com>
References: <CADwseAbUgszMPX0fcDA6Yf6d1GRBa6STMh_zeuEZMs22AS43jQ@mail.gmail.com>
	<CAMYG4GkO1iovSfUrs+ZK8fmoYL+524PCeSBeHwx4E_R9PhZfWw@mail.gmail.com>
Message-ID: <CD664FBB-F84F-46DD-BB8E-40903DA16E92@gmail.com>


On Oct 3, 2013, at 12:24 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 3, 2013 at 2:07 PM, Randall Mackie <rlmackie862 at gmail.com> wrote:
> I am trying to create a VecScatter that will scatter all the values from a horizontal slice of a 3D DMDA global vector (with dof=3) to a sequential vector on every processor. So far I have been unsuccessful, most likely because I don't completely understand how to get the appropriate IS to do this. I believe that you need to create a sequential IS on every processor that has the PETSc global number for the appropriate values I want to scatter. What I most recently tried was the following code which takes the values at level k:
> 
>   DM          :: da3
>   Vec         :: daSlice, daVec
>   PetscInt    :: i, j, k, ierr, nis, istart, istep
>   IS          :: from, to
>   AO          :: ntog
>   VecScatter  :: vscat
>   PetscInt, allocatable :: indices(:)
> 
>   ! Get the AO context for this DM
>   call DMDAGetAO(da3,ntog,ierr)
> 
>   ! Create indices for scatter from global vector
>   allocate (indices(3*nx*ny))
>   istart=(k-1)*nx*ny*3
>   nis=nx*ny*3
>   do i=1,nis
>     indices(i)=istart+i-1
>   end do
> 
>   ! Convert the indices from natural ordering to PETSc ordering
>   call AOApplicationToPetsc(ntog,nis,indices,ierr)
> 
>   call ISCreateGeneral(PETSC_COMM_SELF,nis,indices,PETSC_COPY_VALUES,from,ierr)
> 
>   ! Get the index set that specifies elements to scatter to sequential vector
>   istep=1
>   istart=0
>   nis=3*nx*ny
>   call ISCreateStride(PETSC_COMM_SELF,nis,istart,istep,to,ierr)
> 
>   ! Create the Vector Scatter to get the values to every processor
>   call VecScatterCreate(daVec,from,daSlice,to,vscat,ierr)
> 
>   ! Release memory
>   deallocate (indices)
>   call AODestroy(ntog,ierr)
>   call ISDestroy(from,ierr)
>   call ISDestroy(to,ierr)
> 
> 
> This code is throwing errors as follows
> 
> [2]PETSC ERROR: --------------------- Error Message ------------------------------------
> [2]PETSC ERROR: Invalid argument!
> [2]PETSC ERROR: Wrong type of object: Parameter # 1!
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Libraries linked from /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [2]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [2]PETSC ERROR: Configure options --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich=1
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: AOApplicationToPetsc() line 261 in /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Invalid argument!
> [1]PETSC ERROR: Wrong type of object: Parameter # 1!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Libraries linked from /home/rmackie/PETSc/petsc-3.4.2/linux-gfortran-debug/lib
> [1]PETSC ERROR: Configure run at Thu Sep 19 15:08:47 2013
> [1]PETSC ERROR: Configure options --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich=1
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: AOApplicationToPetsc() line 261 in /home/rmackie/PETSc/petsc-3.4.2/src/vec/is/ao/interface/ao.c
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: -----------------------------------
> 
> etc.
> 
> 
> First, I don't see where my error is that is causing these problems, but second, is there a better way to create this vector scatter without having to use an AO? I suspect there is,  but so far I have been unable to figure out how to create the correct IS.
> 
> I think this is the best way to do what you want. The error says that DMDAGetAO() does not actually give you an AO.
> Can you see what you do get in the debugger? I am wondering if you have overwrite in one of your index arrays.
> 
>   Matt
>  

The error came from my call to AODestroy. Apparently, there is no need to destroy an AO that has been obtained by a call to DMDAGetAO.
Thanks for your help.

Randy


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/1e89c424/attachment.html>

From luqiyue at hotmail.com  Thu Oct  3 21:18:00 2013
From: luqiyue at hotmail.com (Qiyue Lu)
Date: Fri, 4 Oct 2013 02:18:00 +0000
Subject: [petsc-users] Condition number of a matrix
Message-ID: <COL126-W23927AC1D97B6805A1CEF2A7100@phx.gbl>

Dear All:

I am solving a linear system Ax=b with about 40 millions DOF in PETSc. The solver is CR/Bjacobi. The driver is modified from source code : 
petsc-3.3-p6/src/ksp/ksp/examples/tutorials/ex10.c 
I am expecting the condition number of matrix A can be included in the output file but it doesn't now. I am wondering, is there any function or option eligible to print its condition number? thanks. 

Qiyue Lu 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/679f82da/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct  3 21:25:11 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 03 Oct 2013 21:25:11 -0500
Subject: [petsc-users] Condition number of a matrix
In-Reply-To: <COL126-W23927AC1D97B6805A1CEF2A7100@phx.gbl>
References: <COL126-W23927AC1D97B6805A1CEF2A7100@phx.gbl>
Message-ID: <87bo3522ns.fsf@mcs.anl.gov>

Qiyue Lu <luqiyue at hotmail.com> writes:

> Dear All:
>
> I am solving a linear system Ax=b with about 40 millions DOF in PETSc. The solver is CR/Bjacobi. The driver is modified from source code : 
> petsc-3.3-p6/src/ksp/ksp/examples/tutorials/ex10.c 
> I am expecting the condition number of matrix A can be included in the output file but it doesn't now. I am wondering, is there any function or option eligible to print its condition number? thanks. 

Computing an accurate condition number is more expensive than solving a
linear system.  Your best bet if you need an accurate condition number
is to use SLEPc to compute the largest and smallest values.

When using GMRES or CG, you can use -ksp_monitor_singular_value to get
estimates of the largest and smallest singular values of the
*preconditioned* operator.  For large problems, this approach converges
too slowly without preconditioning to be useful for estimating the
extrema of the unpreconditioned operator.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/08175b0b/attachment-0001.pgp>

From luqiyue at hotmail.com  Thu Oct  3 22:53:02 2013
From: luqiyue at hotmail.com (Qiyue Lu)
Date: Fri, 4 Oct 2013 03:53:02 +0000
Subject: [petsc-users] _pc_type
Message-ID: <COL126-W6AACAC16587A2AFCE7DF2A7100@phx.gbl>

Dear All:
When I am solving a linear system. in command line I use:
-ksp_type=cr 
then in the output file: 
I have : job75557 7959 5e+00 2.8e+02 1.0e-01 2.8e+02  bjacobi blks=128 ilu     lvls=0,order=natural

When I am using: 
-ksp_type cr _pc_type bjacobi
the output file looks the same. 

From the driver source code: 
bjacobi blks=128 ilu
is the kspinfo output. 

I am wondering, Does this mean that bjacobi is the default preconditioner for cr? 


Thanks

Qiyue Lu



 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/b3b50600/attachment.html>

From hzhang at mcs.anl.gov  Thu Oct  3 23:09:51 2013
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 3 Oct 2013 23:09:51 -0500
Subject: [petsc-users] _pc_type
In-Reply-To: <COL126-W6AACAC16587A2AFCE7DF2A7100@phx.gbl>
References: <COL126-W6AACAC16587A2AFCE7DF2A7100@phx.gbl>
Message-ID: <CAGCphBtT7QPrL0akYHvvCUGpf5Qgk4yo=kAi=bW-nxDvveB4+g@mail.gmail.com>

Qiyue:
For linear solvers, bjacobi/ilu(0) is the default PETSc preconditioner.
You can change it with option '-pc_type <>', and
use '-ksp_view' to check what solver is being used.

Hong

Dear All:
> When I am solving a linear system. in command line I use:
> -ksp_type=cr
> then in the output file:
> I have : job75557 7959 5e+00 2.8e+02 1.0e-01 2.8e+02  bjacobi blks=128
> ilu     lvls=0,order=natural
>
> When I am using:
> -ksp_type cr _pc_type bjacobi
> the output file looks the same.
>
> From the driver source code:
> bjacobi blks=128 ilu
> is the kspinfo output.
>
> I am wondering, Does this mean that bjacobi is the default preconditioner
> for cr?
>
>
> Thanks
>
> Qiyue Lu
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131003/eb1ecbe7/attachment.html>

From bisheshkh at gmail.com  Fri Oct  4 06:11:11 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 4 Oct 2013 13:11:11 +0200
Subject: [petsc-users] accessing sequential data (e.g. image) at (i, j,
	k) in DM in parallel
Message-ID: <CAEhex8jnk1F08RTkfWYJO9HrU4_b8MhhiDiafFXa2kFTd9D3_Q@mail.gmail.com>

Dear all,

I have a part of a code that does NOT use Petsc, and contains an image K of
dimensions mXnXr.
It also provides me with a function to access any value at (i,j,k), e.g.
using K.getVoxelAt(i,j,k).


Now in my petsc code I create a DMDA of size mXnXr to solve a pde. The
coefficients of the pde at each grid point should physically correspond to
the one in the image K.
The usual code I would use to create a matrix from DM is:

DMDALocalInfo info;
ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);

for (PetscInt k = info.zs; k<info.zs+info.zm; ++k) {
        for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
            for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {

//Here (i,j,k) refers to the indexing/ordering that is internally
organized/distributed to different processors.

            }
        }
 }

Now, the problem is that I cannot use getVoxel(i,j,k) within the above loop
because the ordering and distribution of the image K and the function
getVoxel(i,j,k) knows nothing about the "da" I create in my part of the
code. K might have been stored by each processor locally. So I believe if I
want to use the coefficients from the image K in my code within the above
loop, first I need to either:
1) create global vectors from the DMDA and copy all the values of image K
to this global vector. Then access the values from this global vector.
or
2) find a way to transform (i,j,k) within the above loop to a valid (i,j,k)
for the function getVoxel(i,j,k)

What should I do to resolve the problem ?

Thanks,
Bishesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/fa1ffc09/attachment.html>

From knepley at gmail.com  Fri Oct  4 06:23:02 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Oct 2013 06:23:02 -0500
Subject: [petsc-users] accessing sequential data (e.g. image) at (i, j,
 k) in DM in parallel
In-Reply-To: <CAEhex8jnk1F08RTkfWYJO9HrU4_b8MhhiDiafFXa2kFTd9D3_Q@mail.gmail.com>
References: <CAEhex8jnk1F08RTkfWYJO9HrU4_b8MhhiDiafFXa2kFTd9D3_Q@mail.gmail.com>
Message-ID: <CAMYG4Gk2SreLba7d6ZoMMJqTCh9s6e2V5kCXNWZp2XZvPa-KDA@mail.gmail.com>

On Fri, Oct 4, 2013 at 6:11 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> Dear all,
>
> I have a part of a code that does NOT use Petsc, and contains an image K
> of dimensions mXnXr.
> It also provides me with a function to access any value at (i,j,k), e.g.
> using K.getVoxelAt(i,j,k).
>
>
> Now in my petsc code I create a DMDA of size mXnXr to solve a pde. The
> coefficients of the pde at each grid point should physically correspond to
> the one in the image K.
> The usual code I would use to create a matrix from DM is:
>
> DMDALocalInfo info;
> ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>
> for (PetscInt k = info.zs; k<info.zs+info.zm; ++k) {
>         for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>             for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>
> //Here (i,j,k) refers to the indexing/ordering that is internally
> organized/distributed to different processors.
>
>             }
>         }
>  }
>
> Now, the problem is that I cannot use getVoxel(i,j,k) within the above
> loop because the ordering and distribution of the image K and the function
> getVoxel(i,j,k) knows nothing about the "da" I create in my part of the
> code. K might have been stored by each processor locally. So I believe if I
> want to use the coefficients from the image K in my code within the above
> loop, first I need to either:
> 1) create global vectors from the DMDA and copy all the values of image K
> to this global vector. Then access the values from this global vector.
> or
> 2) find a way to transform (i,j,k) within the above loop to a valid
> (i,j,k) for the function getVoxel(i,j,k)
>
> What should I do to resolve the problem ?
>

This explanation is confused. I do not think you mean "global vector", but
rather you mean the K would be completely held by every process. There
is only one question here:

   Is your image K distributed across processes?

If so, just match your DA to that distribution. If not, its replicated on
every process and getVoxel() will work fine.

   Matt


> Thanks,
> Bishesh
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/0b6af0ab/attachment.html>

From bisheshkh at gmail.com  Fri Oct  4 07:38:05 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 4 Oct 2013 14:38:05 +0200
Subject: [petsc-users] accessing sequential data (e.g. image) at (i, j,
 k) in DM in parallel
In-Reply-To: <CAMYG4Gk2SreLba7d6ZoMMJqTCh9s6e2V5kCXNWZp2XZvPa-KDA@mail.gmail.com>
References: <CAEhex8jnk1F08RTkfWYJO9HrU4_b8MhhiDiafFXa2kFTd9D3_Q@mail.gmail.com>
	<CAMYG4Gk2SreLba7d6ZoMMJqTCh9s6e2V5kCXNWZp2XZvPa-KDA@mail.gmail.com>
Message-ID: <CAEhex8i==3WaFiEQqs3rsvTOJpOCDNPV1a6syEFisDvJSLCA4Q@mail.gmail.com>

On Fri, Oct 4, 2013 at 1:23 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 4, 2013 at 6:11 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> Dear all,
>>
>> I have a part of a code that does NOT use Petsc, and contains an image K
>> of dimensions mXnXr.
>> It also provides me with a function to access any value at (i,j,k), e.g.
>> using K.getVoxelAt(i,j,k).
>>
>>
>> Now in my petsc code I create a DMDA of size mXnXr to solve a pde. The
>> coefficients of the pde at each grid point should physically correspond to
>> the one in the image K.
>> The usual code I would use to create a matrix from DM is:
>>
>> DMDALocalInfo info;
>> ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>
>> for (PetscInt k = info.zs; k<info.zs+info.zm; ++k) {
>>         for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>             for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>
>> //Here (i,j,k) refers to the indexing/ordering that is internally
>> organized/distributed to different processors.
>>
>>             }
>>         }
>>  }
>>
>> Now, the problem is that I cannot use getVoxel(i,j,k) within the above
>> loop because the ordering and distribution of the image K and the function
>> getVoxel(i,j,k) knows nothing about the "da" I create in my part of the
>> code. K might have been stored by each processor locally. So I believe if I
>> want to use the coefficients from the image K in my code within the above
>> loop, first I need to either:
>> 1) create global vectors from the DMDA and copy all the values of image K
>> to this global vector. Then access the values from this global vector.
>> or
>> 2) find a way to transform (i,j,k) within the above loop to a valid
>> (i,j,k) for the function getVoxel(i,j,k)
>>
>> What should I do to resolve the problem ?
>>
>
> This explanation is confused. I do not think you mean "global vector", but
> rather you mean the K would be completely held by every process. There
> is only one question here:
>
>    Is your image K distributed across processes?
>

No, it is not.

>
> If so, just match your DA to that distribution. If not, its replicated on
> every process and getVoxel() will work fine.
>

Yes, thanks it works since K is in every process. Sorry, got confused
(debugging a code for too long makes sometimes people think in weird ways!)


>
>    Matt
>
>
>> Thanks,
>> Bishesh
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/576bd877/attachment-0001.html>

From asmund.ervik at ntnu.no  Fri Oct  4 08:35:35 2013
From: asmund.ervik at ntnu.no (=?ISO-8859-1?Q?=C5smund_Ervik?=)
Date: Fri, 04 Oct 2013 15:35:35 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
Message-ID: <524EC427.9020907@ntnu.no>

Dear all,

We have a two-phase incompressible Navier-Stokes solver written in
Fortran where we use PETSc for solving the pressure Poisson equation.
Since both PETSc and parallelism was an afterthought to this code, it
doesn't scale well at all, so I am tasked with re-writing the whole
thing now. Before I commit any fresh mistakes in the design of this new
code, I will ask for input on my "design decisions" so far.

I want to do domain decomposition on a structured 3D grid. I've been
trying to wrap my head around the DM and DMDA parts of PETSc, and as far
as I understand, these will help me solve the pressure Poisson equation
on a decomposed domain (and with geometric multigrid via Galerkin)
fairly easily.

The tricky part, then; it seems that I must handle "the rest" of the
domain decomposition myself. Omitting some detail, this means my code will:

* set up parameters, initial conditions, etc.
* decompose my array for the velocity field into several parts,
* time loop:
	* communicate e.g. the velocity field on the boundaries
	* each mpi worker will calculate on the local domain the
	  intermediate velocity field, the rhs to the Poisson equation
	  and set up the correct sparse matrix
	* PETSc will solve the Poisson equation to give me the pressure
	* each mpi worker will then calculate the updated
	  divergence-free velocity field
	* each mpi worker will calculate the time step (CFL condition),
	  and we choose the lowest dt among all nodes
* end time loop

Have I misunderstood anything here? At first I thought the DMDA would
give me the framework for decomposing the velocity field, handling
communication of the ghost values at the boundaries etc, but it seems
this is not the case?

One further question: is it a good idea to set up the DMDA letting PETSc
decide the number of processors in each direction, and then using this
same partition for the rest of my code?

If there are any unclear details, please ask. If it matters, I am using
the level-set and ghost-fluid methods, so the matrix for my Poisson
equation must be recomputed each time step. I believe this is the same
situation as Michele Rosso who posted on this list recently.

Best regards,
?smund Ervik

From bsmith at mcs.anl.gov  Fri Oct  4 09:55:49 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 4 Oct 2013 09:55:49 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <524EC427.9020907@ntnu.no>
References: <524EC427.9020907@ntnu.no>
Message-ID: <8916F267-85EA-4A1B-BD43-AA3D57E37D4A@mcs.anl.gov>


   Asmund,

    You can use the DMDA to manage the layout of your velocity variables as well as the pressure variables. You will have two DMDA, one that manages the cell-centered pressure variables (this is created with the dof argument of 1) and one that handles the velocities (that is created with the dof argument of 3) on the "faces". Then you can have a ghosted representation of the velocities from which you compute the right hand side for your pressure equation. 

    What kind of boundary conditions do you have for the velocities? This will determine exactly how to create the DMDA for the velocities. 

    Note the though the x, y, and z velocities are physically associated with the three sets of faces of the cells and thus not collocated on the physical domain you can stack the three of them up at the same i,j,k mesh point of the DMDA vector.  Depending on your boundary conditions there may be less pressure variables then velocity variables in each direction of the grid; to make the two different DMDA "line up" you can just have an extra "slab" of pressure variables in each direction that are never computed on. It's easy to draw a picture in 2d of the stagger grid to see what I mean.


   Barry

On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

> Dear all,
> 
> We have a two-phase incompressible Navier-Stokes solver written in
> Fortran where we use PETSc for solving the pressure Poisson equation.
> Since both PETSc and parallelism was an afterthought to this code, it
> doesn't scale well at all, so I am tasked with re-writing the whole
> thing now. Before I commit any fresh mistakes in the design of this new
> code, I will ask for input on my "design decisions" so far.
> 
> I want to do domain decomposition on a structured 3D grid. I've been
> trying to wrap my head around the DM and DMDA parts of PETSc, and as far
> as I understand, these will help me solve the pressure Poisson equation
> on a decomposed domain (and with geometric multigrid via Galerkin)
> fairly easily.
> 
> The tricky part, then; it seems that I must handle "the rest" of the
> domain decomposition myself. Omitting some detail, this means my code will:
> 
> * set up parameters, initial conditions, etc.
> * decompose my array for the velocity field into several parts,
> * time loop:
> 	* communicate e.g. the velocity field on the boundaries
> 	* each mpi worker will calculate on the local domain the
> 	  intermediate velocity field, the rhs to the Poisson equation
> 	  and set up the correct sparse matrix
> 	* PETSc will solve the Poisson equation to give me the pressure
> 	* each mpi worker will then calculate the updated
> 	  divergence-free velocity field
> 	* each mpi worker will calculate the time step (CFL condition),
> 	  and we choose the lowest dt among all nodes
> * end time loop
> 
> Have I misunderstood anything here? At first I thought the DMDA would
> give me the framework for decomposing the velocity field, handling
> communication of the ghost values at the boundaries etc, but it seems
> this is not the case?
> 
> One further question: is it a good idea to set up the DMDA letting PETSc
> decide the number of processors in each direction, and then using this
> same partition for the rest of my code?
> 
> If there are any unclear details, please ask. If it matters, I am using
> the level-set and ghost-fluid methods, so the matrix for my Poisson
> equation must be recomputed each time step. I believe this is the same
> situation as Michele Rosso who posted on this list recently.
> 
> Best regards,
> ?smund Ervik


From juhaj at iki.fi  Fri Oct  4 10:57:00 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Fri, 04 Oct 2013 17:57 +0200
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <8738q75emc.fsf@mcs.anl.gov>
References: <2569883.TWAWmgZMUo@rigel> <13652175.rAY5a1HGtb@rigel>
	<8738q75emc.fsf@mcs.anl.gov>
Message-ID: <14240863.L6qv0C0M0G@dhcp071>

On Sunday 18 August 2013 08:10:19 Jed Brown wrote:
> Output uses a collective write, so the granularity of the IO node is
> probably more relevant for writing (e.g., BG/Q would have one IO node
> per 128 compute nodes), but almost any chunk size should perform
> similarly.  It would make a lot more difference for something like

I ran into this on a Cray XK30 and it's certainly not the case there that any 
chunk size performs even close to similarly: I can get IO throughput from 
roughly 50 MB/s to 16 GB/s depending on the chunk sizes and number of ranks 
participating in the MPI IO operation (underneath H5Dwrite()). Unfortunately, 
there is still the lustre stripes to consider, which might drastically change 
things, so I doubt there is a good overall choice here.

> visualization where subsets of the data are read, typically with
> independent IO.

Yes, this certainly needs to be considered, too. I guess huge chunks are bad 
here?

> Are you sure?  Did you try writing a second time step?  The
> documentation says that H5S_UNLIMITED requires chunking.

Sorry, I didn't consider several timesteps and extending the files at all. 
You're correct, of course.

> Chunk size needs to be collective.  We could compute an average size
> From each subdomain, but can't just use the subdomain size.

Why not use the size of the local part of the DA/Vec? That would guarantee

1. That the chunk is not ridiculously large (unless someone runs on a huge SMP 
with a single thread, which is probably something no one should be doing 
anyway).

2. Whatever gets written to disc is contiguous both on disc and in memory, 
eliminating unnecessary seek()s and cache thrashing (although the latter is 
probably irrelevant as the disc is going to be the bottleneck).

3. The chunk is most likely < 4 GB as very few machines have more than 4 GB / 
core available. (Well, this is not really a guarantee.)

> I think the chunk size (or maximum chunk size) should be settable by the
> user.

I agree, that would be the best solution.

Is the granularity (number of ranks actually doing disc IO) settable on HDF5 
side or does that need to be set in MPI-IO?

Any idea which version of PETSc this fix might get into? I currently keep my 
own patched version of gr2.c around, which uses local-Vec-size chunks and it 
works ok, but I'd like to be able to use vanilla PETSc again.

Cheers,
Juha


From asmund.ervik at ntnu.no  Fri Oct  4 12:57:40 2013
From: asmund.ervik at ntnu.no (=?iso-8859-1?Q?=C5smund_Ervik?=)
Date: Fri, 4 Oct 2013 17:57:40 +0000
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <8916F267-85EA-4A1B-BD43-AA3D57E37D4A@mcs.anl.gov>
References: <524EC427.9020907@ntnu.no>,
	<8916F267-85EA-4A1B-BD43-AA3D57E37D4A@mcs.anl.gov>
Message-ID: <jxeljhlbwvlc82t1ve51onm1.1380909453613@email.android.com>

Barry,

Thanks for the quick answer.

Good to hear that I can use the DMDA framework for all variables. Should I put all scalars (e.g. pressure, level set function, etc) in the same DA, or should I keep a distinct one for the pressure (where I want to use multigrid)?

The reason I was unsure is that I can't seem to find an example which manipulates the local array from a DA. I would've guessed there was something like

real, dimension(:,:,:) u,v,w
call DMDAGetLocalArray(da,u,v,w)
! Some computations looping over local i,j,k that manipulate u,v,w
call DMDARestoreLocalArray(da,u,v,w)

On the BCs for velocity, I would like to support several options. To get the code up and running I would be OK with just periodic, but I would eventually like to support full slip and no slip, and preferably a mix of these for the different faces. Perhaps also inflow and outflow. I don't need (physical) pressure BCs though.  Would this complicate things much?

I understand the point about the velocity i,j,k lining up, this is how we do it currently.

?smund

Sent from my VT-102

Barry Smith <bsmith at mcs.anl.gov> skrev:

   Asmund,

    You can use the DMDA to manage the layout of your velocity variables as well as the pressure variables. You will have two DMDA, one that manages the cell-centered pressure variables (this is created with the dof argument of 1) and one that handles the velocities (that is created with the dof argument of 3) on the "faces". Then you can have a ghosted representation of the velocities from which you compute the right hand side for your pressure equation.

    What kind of boundary conditions do you have for the velocities? This will determine exactly how to create the DMDA for the velocities.

    Note the though the x, y, and z velocities are physically associated with the three sets of faces of the cells and thus not collocated on the physical domain you can stack the three of them up at the same i,j,k mesh point of the DMDA vector.  Depending on your boundary conditions there may be less pressure variables then velocity variables in each direction of the grid; to make the two different DMDA "line up" you can just have an extra "slab" of pressure variables in each direction that are never computed on. It's easy to draw a picture in 2d of the stagger grid to see what I mean.


   Barry

On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

> Dear all,
>
> We have a two-phase incompressible Navier-Stokes solver written in
> Fortran where we use PETSc for solving the pressure Poisson equation.
> Since both PETSc and parallelism was an afterthought to this code, it
> doesn't scale well at all, so I am tasked with re-writing the whole
> thing now. Before I commit any fresh mistakes in the design of this new
> code, I will ask for input on my "design decisions" so far.
>
> I want to do domain decomposition on a structured 3D grid. I've been
> trying to wrap my head around the DM and DMDA parts of PETSc, and as far
> as I understand, these will help me solve the pressure Poisson equation
> on a decomposed domain (and with geometric multigrid via Galerkin)
> fairly easily.
>
> The tricky part, then; it seems that I must handle "the rest" of the
> domain decomposition myself. Omitting some detail, this means my code will:
>
> * set up parameters, initial conditions, etc.
> * decompose my array for the velocity field into several parts,
> * time loop:
>        * communicate e.g. the velocity field on the boundaries
>        * each mpi worker will calculate on the local domain the
>          intermediate velocity field, the rhs to the Poisson equation
>          and set up the correct sparse matrix
>        * PETSc will solve the Poisson equation to give me the pressure
>        * each mpi worker will then calculate the updated
>          divergence-free velocity field
>        * each mpi worker will calculate the time step (CFL condition),
>          and we choose the lowest dt among all nodes
> * end time loop
>
> Have I misunderstood anything here? At first I thought the DMDA would
> give me the framework for decomposing the velocity field, handling
> communication of the ghost values at the boundaries etc, but it seems
> this is not the case?
>
> One further question: is it a good idea to set up the DMDA letting PETSc
> decide the number of processors in each direction, and then using this
> same partition for the rest of my code?
>
> If there are any unclear details, please ask. If it matters, I am using
> the level-set and ghost-fluid methods, so the matrix for my Poisson
> equation must be recomputed each time step. I believe this is the same
> situation as Michele Rosso who posted on this list recently.
>
> Best regards,
> ?smund Ervik

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/03dca334/attachment.html>

From knepley at gmail.com  Fri Oct  4 13:03:07 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Oct 2013 13:03:07 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <jxeljhlbwvlc82t1ve51onm1.1380909453613@email.android.com>
References: <524EC427.9020907@ntnu.no>
	<8916F267-85EA-4A1B-BD43-AA3D57E37D4A@mcs.anl.gov>
	<jxeljhlbwvlc82t1ve51onm1.1380909453613@email.android.com>
Message-ID: <CAMYG4G=VSOosf-Dd6j-=SvvJjch=RcrEs-SvbCAZgbyQNVS4tA@mail.gmail.com>

On Fri, Oct 4, 2013 at 12:57 PM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

>  Barry,
>
>  Thanks for the quick answer.
>
>  Good to hear that I can use the DMDA framework for all variables. Should
> I put all scalars (e.g. pressure, level set function, etc) in the same DA,
> or should I keep a distinct one for the pressure (where I want to use
> multigrid)?
>

Separate variables which are solved for.


> The reason I was unsure is that I can't seem to find an example which
> manipulates the local array from a DA. I would've guessed there was
> something like
>
>  real, dimension(:,:,:) u,v,w
> call DMDAGetLocalArray(da,u,v,w)
> ! Some computations looping over local i,j,k that manipulate u,v,w
> call DMDARestoreLocalArray(da,u,v,w)
>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html


> On the BCs for velocity, I would like to support several options. To get
> the code up and running I would be OK with just periodic, but I would
> eventually like to support full slip and no slip, and preferably a mix of
> these for the different faces. Perhaps also inflow and outflow. I don't
> need (physical) pressure BCs though.  Would this complicate things much?
>

Both periodic and ghost cells are supported. Imposing Dirichlet conditions
on an unknown is also easy.

  Thanks,

    Matt


> I understand the point about the velocity i,j,k lining up, this is how we
> do it currently.
>
>  ?smund
>
>  Sent from my VT-102
>
> Barry Smith <bsmith at mcs.anl.gov> skrev:
>
>    Asmund,
>
>     You can use the DMDA to manage the layout of your velocity variables
> as well as the pressure variables. You will have two DMDA, one that manages
> the cell-centered pressure variables (this is created with the dof argument
> of 1) and one that handles the velocities (that is created with the dof
> argument of 3) on the "faces". Then you can have a ghosted representation
> of the velocities from which you compute the right hand side for your
> pressure equation.
>
>     What kind of boundary conditions do you have for the velocities? This
> will determine exactly how to create the DMDA for the velocities.
>
>     Note the though the x, y, and z velocities are physically associated
> with the three sets of faces of the cells and thus not collocated on the
> physical domain you can stack the three of them up at the same i,j,k mesh
> point of the DMDA vector.  Depending on your boundary conditions there may
> be less pressure variables then velocity variables in each direction of the
> grid; to make the two different DMDA "line up" you can just have an extra
> "slab" of pressure variables in each direction that are never computed on.
> It's easy to draw a picture in 2d of the stagger grid to see what I mean.
>
>
>    Barry
>
> On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
>
> > Dear all,
> >
> > We have a two-phase incompressible Navier-Stokes solver written in
> > Fortran where we use PETSc for solving the pressure Poisson equation.
> > Since both PETSc and parallelism was an afterthought to this code, it
> > doesn't scale well at all, so I am tasked with re-writing the whole
> > thing now. Before I commit any fresh mistakes in the design of this new
> > code, I will ask for input on my "design decisions" so far.
> >
> > I want to do domain decomposition on a structured 3D grid. I've been
> > trying to wrap my head around the DM and DMDA parts of PETSc, and as far
> > as I understand, these will help me solve the pressure Poisson equation
> > on a decomposed domain (and with geometric multigrid via Galerkin)
> > fairly easily.
> >
> > The tricky part, then; it seems that I must handle "the rest" of the
> > domain decomposition myself. Omitting some detail, this means my code
> will:
> >
> > * set up parameters, initial conditions, etc.
> > * decompose my array for the velocity field into several parts,
> > * time loop:
> >        * communicate e.g. the velocity field on the boundaries
> >        * each mpi worker will calculate on the local domain the
> >          intermediate velocity field, the rhs to the Poisson equation
> >          and set up the correct sparse matrix
> >        * PETSc will solve the Poisson equation to give me the pressure
> >        * each mpi worker will then calculate the updated
> >          divergence-free velocity field
> >        * each mpi worker will calculate the time step (CFL condition),
> >          and we choose the lowest dt among all nodes
> > * end time loop
> >
> > Have I misunderstood anything here? At first I thought the DMDA would
> > give me the framework for decomposing the velocity field, handling
> > communication of the ghost values at the boundaries etc, but it seems
> > this is not the case?
> >
> > One further question: is it a good idea to set up the DMDA letting PETSc
> > decide the number of processors in each direction, and then using this
> > same partition for the rest of my code?
> >
> > If there are any unclear details, please ask. If it matters, I am using
> > the level-set and ghost-fluid methods, so the matrix for my Poisson
> > equation must be recomputed each time step. I believe this is the same
> > situation as Michele Rosso who posted on this list recently.
> >
> > Best regards,
> > ?smund Ervik
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/141a305f/attachment.html>

From pengxwang at hotmail.com  Fri Oct  4 13:48:43 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Fri, 4 Oct 2013 13:48:43 -0500
Subject: [petsc-users] cannot find PetscViewerHDF5 functions in libraries
Message-ID: <BAY179-W5063FBF92A7DF84722E739B4100@phx.gbl>

Hello,

   The version of Petsc I am using is 3.3-p6.  I have hdf5 1.8.11 installed as well as. I am trying to use PetscViewerHDF5Open(), PetscViewerHDF5PushGroup(). But the compiling error shows undefined reference to `PetscViewerHDF5Open' and  undefined reference to `PetscViewerHDF5PushGroup'.  I also tried to compile /petsc-3.3-p6/src/vec/vec/examples/tutorials/ex19.c with petsc style makefile, the same error occurs.

 My config options are --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich --with-cxx=g++ --download-hypre=1 --download-ml=1. Should I re-install petsc with --download-hdf5=1? Thanks.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/51ccfa6f/attachment.html>

From asmund.ervik at ntnu.no  Fri Oct  4 13:51:20 2013
From: asmund.ervik at ntnu.no (=?utf-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Fri, 4 Oct 2013 18:51:20 +0000
Subject: [petsc-users] Writing a domain decomposition code with PETSc
Message-ID: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>


Matthew Knepley <knepley at gmail.com> skrev:
On Fri, Oct 4, 2013 at 12:57 PM, ?smund Ervik <asmund.ervik at ntnu.no<mailto:asmund.ervik at ntnu.no>> wrote:
Barry,

Thanks for the quick answer.

Good to hear that I can use the DMDA framework for all variables. Should I put all scalars (e.g. pressure, level set function, etc) in the same DA, or should I keep a distinct one for the pressure (where I want to use multigrid)?

Separate variables which are solved for.

Ok. So only variables that belong together, such as the velocity components, should be grouped via the dof?

The reason I was unsure is that I can't seem to find an example which manipulates the local array from a DA. I would've guessed there was something like

real, dimension(:,:,:) u,v,w
call DMDAGetLocalArray(da,u,v,w)
! Some computations looping over local i,j,k that manipulate u,v,w
call DMDARestoreLocalArray(da,u,v,w)

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html

Doh. Thanks. I've been looking at that, and kept focusing on the input which must be a vector, but I see that's just the internal storage format.

Speaking of storage formats, I would like to have a unified format for checkpointing and visualization output. So that after analyzing a result I could restart from an arbitrary time step in the output. Is the VTK viewer or the binary PETSc viewer the best way to go? I use Tecplot and Visit today.

On the BCs for velocity, I would like to support several options. To get the code up and running I would be OK with just periodic, but I would eventually like to support full slip and no slip, and preferably a mix of these for the different faces. Perhaps also inflow and outflow. I don't need (physical) pressure BCs though.  Would this complicate things much?

Both periodic and ghost cells are supported. Imposing Dirichlet conditions on an unknown is also easy.

Good. I will come back to this if I have specific questions.

?smund


  Thanks,

    Matt

I understand the point about the velocity i,j,k lining up, this is how we do it currently.

?smund

Sent from my VT-102

Barry Smith <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>> skrev:

   Asmund,

    You can use the DMDA to manage the layout of your velocity variables as well as the pressure variables. You will have two DMDA, one that manages the cell-centered pressure variables (this is created with the dof argument of 1) and one that handles the velocities (that is created with the dof argument of 3) on the "faces". Then you can have a ghosted representation of the velocities from which you compute the right hand side for your pressure equation.

    What kind of boundary conditions do you have for the velocities? This will determine exactly how to create the DMDA for the velocities.

    Note the though the x, y, and z velocities are physically associated with the three sets of faces of the cells and thus not collocated on the physical domain you can stack the three of them up at the same i,j,k mesh point of the DMDA vector.  Depending on your boundary conditions there may be less pressure variables then velocity variables in each direction of the grid; to make the two different DMDA "line up" you can just have an extra "slab" of pressure variables in each direction that are never computed on. It's easy to draw a picture in 2d of the stagger grid to see what I mean.


   Barry

On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no<mailto:asmund.ervik at ntnu.no>> wrote:

> Dear all,
>
> We have a two-phase incompressible Navier-Stokes solver written in
> Fortran where we use PETSc for solving the pressure Poisson equation.
> Since both PETSc and parallelism was an afterthought to this code, it
> doesn't scale well at all, so I am tasked with re-writing the whole
> thing now. Before I commit any fresh mistakes in the design of this new
> code, I will ask for input on my "design decisions" so far.
>
> I want to do domain decomposition on a structured 3D grid. I've been
> trying to wrap my head around the DM and DMDA parts of PETSc, and as far
> as I understand, these will help me solve the pressure Poisson equation
> on a decomposed domain (and with geometric multigrid via Galerkin)
> fairly easily.
>
> The tricky part, then; it seems that I must handle "the rest" of the
> domain decomposition myself. Omitting some detail, this means my code will:
>
> * set up parameters, initial conditions, etc.
> * decompose my array for the velocity field into several parts,
> * time loop:
>        * communicate e.g. the velocity field on the boundaries
>        * each mpi worker will calculate on the local domain the
>          intermediate velocity field, the rhs to the Poisson equation
>          and set up the correct sparse matrix
>        * PETSc will solve the Poisson equation to give me the pressure
>        * each mpi worker will then calculate the updated
>          divergence-free velocity field
>        * each mpi worker will calculate the time step (CFL condition),
>          and we choose the lowest dt among all nodes
> * end time loop
>
> Have I misunderstood anything here? At first I thought the DMDA would
> give me the framework for decomposing the velocity field, handling
> communication of the ghost values at the boundaries etc, but it seems
> this is not the case?
>
> One further question: is it a good idea to set up the DMDA letting PETSc
> decide the number of processors in each direction, and then using this
> same partition for the rest of my code?
>
> If there are any unclear details, please ask. If it matters, I am using
> the level-set and ghost-fluid methods, so the matrix for my Poisson
> equation must be recomputed each time step. I believe this is the same
> situation as Michele Rosso who posted on this list recently.
>
> Best regards,
> ?smund Ervik




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/6e7e3a91/attachment-0001.html>

From knepley at gmail.com  Fri Oct  4 14:00:09 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Oct 2013 14:00:09 -0500
Subject: [petsc-users] cannot find PetscViewerHDF5 functions in libraries
In-Reply-To: <BAY179-W5063FBF92A7DF84722E739B4100@phx.gbl>
References: <BAY179-W5063FBF92A7DF84722E739B4100@phx.gbl>
Message-ID: <CAMYG4G=zWfY3wC_MjxUZfus18wRhAK8EyX19=P8bCwV7vKfbPg@mail.gmail.com>

On Fri, Oct 4, 2013 at 1:48 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Hello,
>
>    The version of Petsc I am using is 3.3-p6.  I have hdf5 1.8.11
> installed as well as. I am trying to use PetscViewerHDF5Open(),
> PetscViewerHDF5PushGroup(). But the compiling error shows undefined
> reference to `PetscViewerHDF5Open' and  undefined reference to
> `PetscViewerHDF5PushGroup'.  I also tried to compile
> /petsc-3.3-p6/src/vec/vec/examples/tutorials/ex19.c with petsc style
> makefile, the same error occurs.
>
>  My config options are --with-cc=gcc --with-fc=gfortran
> --download-f-blas-lapack --download-mpich --with-cxx=g++ --download-hypre=1
> --download-ml=1. Should I re-install petsc with --download-hdf5=1? Thanks.
>

If we do not install HDF5 for you, you still have to activate it during
configure, --with-hdf5-include= --with-hdf5-lib=.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/d304df31/attachment.html>

From knepley at gmail.com  Fri Oct  4 14:04:02 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Oct 2013 14:04:02 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
Message-ID: <CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>

On Fri, Oct 4, 2013 at 1:51 PM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

>
> Matthew Knepley <knepley at gmail.com> skrev:
> On Fri, Oct 4, 2013 at 12:57 PM, ?smund Ervik <asmund.ervik at ntnu.no>wrote:
>
>>  Barry,
>>
>>  Thanks for the quick answer.
>>
>>  Good to hear that I can use the DMDA framework for all variables.
>> Should I put all scalars (e.g. pressure, level set function, etc) in the
>> same DA, or should I keep a distinct one for the pressure (where I want to
>> use multigrid)?
>>
>
>  Separate variables which are solved for.
>
>  Ok. So only variables that belong together, such as the velocity
> components, should be grouped via the dof?
>

Its just the solved for/not solved for distinction that is important.


>   The reason I was unsure is that I can't seem to find an example which
>> manipulates the local array from a DA. I would've guessed there was
>> something like
>>
>>  real, dimension(:,:,:) u,v,w
>> call DMDAGetLocalArray(da,u,v,w)
>> ! Some computations looping over local i,j,k that manipulate u,v,w
>> call DMDARestoreLocalArray(da,u,v,w)
>>
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html
>
>  Doh. Thanks. I've been looking at that, and kept focusing on the input
> which must be a vector, but I see that's just the internal storage format.
>
>  Speaking of storage formats, I would like to have a unified format for
> checkpointing and visualization output. So that after analyzing a result I
> could restart from an arbitrary time step in the output. Is the VTK viewer
> or the binary PETSc viewer the best way to go? I use Tecplot and Visit
> today.
>

Right now, this does not exist. The binary format is great for
checkpointing, but not for viz. In the future, I predict HDF5, since you
can put Xdmf around it (could also do this for PETSc binary) for viz and it
has nice Python tools. Jed probably has an opinion.

   Matt


>
>
>>  On the BCs for velocity, I would like to support several options. To
>> get the code up and running I would be OK with just periodic, but I would
>> eventually like to support full slip and no slip, and preferably a mix of
>> these for the different faces. Perhaps also inflow and outflow. I don't
>> need (physical) pressure BCs though.  Would this complicate things much?
>>
>
>  Both periodic and ghost cells are supported. Imposing Dirichlet
> conditions on an unknown is also easy.
>
>  Good. I will come back to this if I have specific questions.
>
>  ?smund
>
>
>    Thanks,
>
>      Matt
>
>
>>  I understand the point about the velocity i,j,k lining up, this is how
>> we do it currently.
>>
>>  ?smund
>>
>>  Sent from my VT-102
>>
>> Barry Smith <bsmith at mcs.anl.gov> skrev:
>>
>>    Asmund,
>>
>>     You can use the DMDA to manage the layout of your velocity variables
>> as well as the pressure variables. You will have two DMDA, one that manages
>> the cell-centered pressure variables (this is created with the dof argument
>> of 1) and one that handles the velocities (that is created with the dof
>> argument of 3) on the "faces". Then you can have a ghosted representation
>> of the velocities from which you compute the right hand side for your
>> pressure equation.
>>
>>     What kind of boundary conditions do you have for the velocities? This
>> will determine exactly how to create the DMDA for the velocities.
>>
>>     Note the though the x, y, and z velocities are physically associated
>> with the three sets of faces of the cells and thus not collocated on the
>> physical domain you can stack the three of them up at the same i,j,k mesh
>> point of the DMDA vector.  Depending on your boundary conditions there may
>> be less pressure variables then velocity variables in each direction of the
>> grid; to make the two different DMDA "line up" you can just have an extra
>> "slab" of pressure variables in each direction that are never computed on.
>> It's easy to draw a picture in 2d of the stagger grid to see what I mean.
>>
>>
>>    Barry
>>
>> On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
>>
>> > Dear all,
>> >
>> > We have a two-phase incompressible Navier-Stokes solver written in
>> > Fortran where we use PETSc for solving the pressure Poisson equation.
>> > Since both PETSc and parallelism was an afterthought to this code, it
>> > doesn't scale well at all, so I am tasked with re-writing the whole
>> > thing now. Before I commit any fresh mistakes in the design of this new
>> > code, I will ask for input on my "design decisions" so far.
>> >
>> > I want to do domain decomposition on a structured 3D grid. I've been
>> > trying to wrap my head around the DM and DMDA parts of PETSc, and as far
>> > as I understand, these will help me solve the pressure Poisson equation
>> > on a decomposed domain (and with geometric multigrid via Galerkin)
>> > fairly easily.
>> >
>> > The tricky part, then; it seems that I must handle "the rest" of the
>> > domain decomposition myself. Omitting some detail, this means my code
>> will:
>> >
>> > * set up parameters, initial conditions, etc.
>> > * decompose my array for the velocity field into several parts,
>> > * time loop:
>> >        * communicate e.g. the velocity field on the boundaries
>> >        * each mpi worker will calculate on the local domain the
>> >          intermediate velocity field, the rhs to the Poisson equation
>> >          and set up the correct sparse matrix
>> >        * PETSc will solve the Poisson equation to give me the pressure
>> >        * each mpi worker will then calculate the updated
>> >          divergence-free velocity field
>> >        * each mpi worker will calculate the time step (CFL condition),
>> >          and we choose the lowest dt among all nodes
>> > * end time loop
>> >
>> > Have I misunderstood anything here? At first I thought the DMDA would
>> > give me the framework for decomposing the velocity field, handling
>> > communication of the ghost values at the boundaries etc, but it seems
>> > this is not the case?
>> >
>> > One further question: is it a good idea to set up the DMDA letting PETSc
>> > decide the number of processors in each direction, and then using this
>> > same partition for the rest of my code?
>> >
>> > If there are any unclear details, please ask. If it matters, I am using
>> > the level-set and ghost-fluid methods, so the matrix for my Poisson
>> > equation must be recomputed each time step. I believe this is the same
>> > situation as Michele Rosso who posted on this list recently.
>> >
>> > Best regards,
>> > ?smund Ervik
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/020115aa/attachment.html>

From asmund.ervik at ntnu.no  Fri Oct  4 14:10:11 2013
From: asmund.ervik at ntnu.no (=?iso-8859-1?Q?=C5smund_Ervik?=)
Date: Fri, 4 Oct 2013 19:10:11 +0000
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>,
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
Message-ID: <ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>

Is there any reason I should prefer Xdmf over Cgns? I think both use hdf5 in the background.

Apologies by the way for my phone's inability to reply inline properly.

?smund


Sent from my VT-102

Matthew Knepley <knepley at gmail.com> skrev:
On Fri, Oct 4, 2013 at 1:51 PM, ?smund Ervik <asmund.ervik at ntnu.no<mailto:asmund.ervik at ntnu.no>> wrote:

Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> skrev:
On Fri, Oct 4, 2013 at 12:57 PM, ?smund Ervik <asmund.ervik at ntnu.no<mailto:asmund.ervik at ntnu.no>> wrote:
Barry,

Thanks for the quick answer.

Good to hear that I can use the DMDA framework for all variables. Should I put all scalars (e.g. pressure, level set function, etc) in the same DA, or should I keep a distinct one for the pressure (where I want to use multigrid)?

Separate variables which are solved for.

Ok. So only variables that belong together, such as the velocity components, should be grouped via the dof?

Its just the solved for/not solved for distinction that is important.

The reason I was unsure is that I can't seem to find an example which manipulates the local array from a DA. I would've guessed there was something like

real, dimension(:,:,:) u,v,w
call DMDAGetLocalArray(da,u,v,w)
! Some computations looping over local i,j,k that manipulate u,v,w
call DMDARestoreLocalArray(da,u,v,w)

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html

Doh. Thanks. I've been looking at that, and kept focusing on the input which must be a vector, but I see that's just the internal storage format.

Speaking of storage formats, I would like to have a unified format for checkpointing and visualization output. So that after analyzing a result I could restart from an arbitrary time step in the output. Is the VTK viewer or the binary PETSc viewer the best way to go? I use Tecplot and Visit today.

Right now, this does not exist. The binary format is great for checkpointing, but not for viz. In the future, I predict HDF5, since you
can put Xdmf around it (could also do this for PETSc binary) for viz and it has nice Python tools. Jed probably has an opinion.

   Matt


On the BCs for velocity, I would like to support several options. To get the code up and running I would be OK with just periodic, but I would eventually like to support full slip and no slip, and preferably a mix of these for the different faces. Perhaps also inflow and outflow. I don't need (physical) pressure BCs though.  Would this complicate things much?

Both periodic and ghost cells are supported. Imposing Dirichlet conditions on an unknown is also easy.

Good. I will come back to this if I have specific questions.

?smund


  Thanks,

    Matt

I understand the point about the velocity i,j,k lining up, this is how we do it currently.

?smund

Sent from my VT-102

Barry Smith <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>> skrev:

   Asmund,

    You can use the DMDA to manage the layout of your velocity variables as well as the pressure variables. You will have two DMDA, one that manages the cell-centered pressure variables (this is created with the dof argument of 1) and one that handles the velocities (that is created with the dof argument of 3) on the "faces". Then you can have a ghosted representation of the velocities from which you compute the right hand side for your pressure equation.

    What kind of boundary conditions do you have for the velocities? This will determine exactly how to create the DMDA for the velocities.

    Note the though the x, y, and z velocities are physically associated with the three sets of faces of the cells and thus not collocated on the physical domain you can stack the three of them up at the same i,j,k mesh point of the DMDA vector.  Depending on your boundary conditions there may be less pressure variables then velocity variables in each direction of the grid; to make the two different DMDA "line up" you can just have an extra "slab" of pressure variables in each direction that are never computed on. It's easy to draw a picture in 2d of the stagger grid to see what I mean.


   Barry

On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no<mailto:asmund.ervik at ntnu.no>> wrote:

> Dear all,
>
> We have a two-phase incompressible Navier-Stokes solver written in
> Fortran where we use PETSc for solving the pressure Poisson equation.
> Since both PETSc and parallelism was an afterthought to this code, it
> doesn't scale well at all, so I am tasked with re-writing the whole
> thing now. Before I commit any fresh mistakes in the design of this new
> code, I will ask for input on my "design decisions" so far.
>
> I want to do domain decomposition on a structured 3D grid. I've been
> trying to wrap my head around the DM and DMDA parts of PETSc, and as far
> as I understand, these will help me solve the pressure Poisson equation
> on a decomposed domain (and with geometric multigrid via Galerkin)
> fairly easily.
>
> The tricky part, then; it seems that I must handle "the rest" of the
> domain decomposition myself. Omitting some detail, this means my code will:
>
> * set up parameters, initial conditions, etc.
> * decompose my array for the velocity field into several parts,
> * time loop:
>        * communicate e.g. the velocity field on the boundaries
>        * each mpi worker will calculate on the local domain the
>          intermediate velocity field, the rhs to the Poisson equation
>          and set up the correct sparse matrix
>        * PETSc will solve the Poisson equation to give me the pressure
>        * each mpi worker will then calculate the updated
>          divergence-free velocity field
>        * each mpi worker will calculate the time step (CFL condition),
>          and we choose the lowest dt among all nodes
> * end time loop
>
> Have I misunderstood anything here? At first I thought the DMDA would
> give me the framework for decomposing the velocity field, handling
> communication of the ghost values at the boundaries etc, but it seems
> this is not the case?
>
> One further question: is it a good idea to set up the DMDA letting PETSc
> decide the number of processors in each direction, and then using this
> same partition for the rest of my code?
>
> If there are any unclear details, please ask. If it matters, I am using
> the level-set and ghost-fluid methods, so the matrix for my Poisson
> equation must be recomputed each time step. I believe this is the same
> situation as Michele Rosso who posted on this list recently.
>
> Best regards,
> ?smund Ervik




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/01e8bd03/attachment-0001.html>

From knepley at gmail.com  Fri Oct  4 14:22:21 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 Oct 2013 14:22:21 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
Message-ID: <CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>

On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

>  Is there any reason I should prefer Xdmf over Cgns? I think both use
> hdf5 in the background.
>

I guess not. It was just easier for me to get Xdmf going. We have trial
CGNS as well.

  Matt


> Apologies by the way for my phone's inability to reply inline properly.
>
>  ?smund
>
>
>  Sent from my VT-102
>
> Matthew Knepley <knepley at gmail.com> skrev:
>  On Fri, Oct 4, 2013 at 1:51 PM, ?smund Ervik <asmund.ervik at ntnu.no>wrote:
>
>>
>> Matthew Knepley <knepley at gmail.com> skrev:
>> On Fri, Oct 4, 2013 at 12:57 PM, ?smund Ervik <asmund.ervik at ntnu.no>wrote:
>>
>>>  Barry,
>>>
>>>  Thanks for the quick answer.
>>>
>>>  Good to hear that I can use the DMDA framework for all variables.
>>> Should I put all scalars (e.g. pressure, level set function, etc) in the
>>> same DA, or should I keep a distinct one for the pressure (where I want to
>>> use multigrid)?
>>>
>>
>>  Separate variables which are solved for.
>>
>>  Ok. So only variables that belong together, such as the velocity
>> components, should be grouped via the dof?
>>
>
>  Its just the solved for/not solved for distinction that is important.
>
>
>>     The reason I was unsure is that I can't seem to find an example
>>> which manipulates the local array from a DA. I would've guessed there was
>>> something like
>>>
>>>  real, dimension(:,:,:) u,v,w
>>> call DMDAGetLocalArray(da,u,v,w)
>>> ! Some computations looping over local i,j,k that manipulate u,v,w
>>> call DMDARestoreLocalArray(da,u,v,w)
>>>
>>
>>
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html
>>
>>  Doh. Thanks. I've been looking at that, and kept focusing on the input
>> which must be a vector, but I see that's just the internal storage format.
>>
>>  Speaking of storage formats, I would like to have a unified format for
>> checkpointing and visualization output. So that after analyzing a result I
>> could restart from an arbitrary time step in the output. Is the VTK viewer
>> or the binary PETSc viewer the best way to go? I use Tecplot and Visit
>> today.
>>
>
>  Right now, this does not exist. The binary format is great for
> checkpointing, but not for viz. In the future, I predict HDF5, since you
> can put Xdmf around it (could also do this for PETSc binary) for viz and
> it has nice Python tools. Jed probably has an opinion.
>
>     Matt
>
>
>>
>>
>>>  On the BCs for velocity, I would like to support several options. To
>>> get the code up and running I would be OK with just periodic, but I would
>>> eventually like to support full slip and no slip, and preferably a mix of
>>> these for the different faces. Perhaps also inflow and outflow. I don't
>>> need (physical) pressure BCs though.  Would this complicate things much?
>>>
>>
>>  Both periodic and ghost cells are supported. Imposing Dirichlet
>> conditions on an unknown is also easy.
>>
>>  Good. I will come back to this if I have specific questions.
>>
>>  ?smund
>>
>>
>>    Thanks,
>>
>>      Matt
>>
>>
>>>  I understand the point about the velocity i,j,k lining up, this is how
>>> we do it currently.
>>>
>>>  ?smund
>>>
>>>  Sent from my VT-102
>>>
>>> Barry Smith <bsmith at mcs.anl.gov> skrev:
>>>
>>>    Asmund,
>>>
>>>     You can use the DMDA to manage the layout of your velocity variables
>>> as well as the pressure variables. You will have two DMDA, one that manages
>>> the cell-centered pressure variables (this is created with the dof argument
>>> of 1) and one that handles the velocities (that is created with the dof
>>> argument of 3) on the "faces". Then you can have a ghosted representation
>>> of the velocities from which you compute the right hand side for your
>>> pressure equation.
>>>
>>>     What kind of boundary conditions do you have for the velocities?
>>> This will determine exactly how to create the DMDA for the velocities.
>>>
>>>     Note the though the x, y, and z velocities are physically associated
>>> with the three sets of faces of the cells and thus not collocated on the
>>> physical domain you can stack the three of them up at the same i,j,k mesh
>>> point of the DMDA vector.  Depending on your boundary conditions there may
>>> be less pressure variables then velocity variables in each direction of the
>>> grid; to make the two different DMDA "line up" you can just have an extra
>>> "slab" of pressure variables in each direction that are never computed on.
>>> It's easy to draw a picture in 2d of the stagger grid to see what I mean.
>>>
>>>
>>>    Barry
>>>
>>> On Oct 4, 2013, at 8:35 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
>>>
>>> > Dear all,
>>> >
>>> > We have a two-phase incompressible Navier-Stokes solver written in
>>> > Fortran where we use PETSc for solving the pressure Poisson equation.
>>> > Since both PETSc and parallelism was an afterthought to this code, it
>>> > doesn't scale well at all, so I am tasked with re-writing the whole
>>> > thing now. Before I commit any fresh mistakes in the design of this new
>>> > code, I will ask for input on my "design decisions" so far.
>>> >
>>> > I want to do domain decomposition on a structured 3D grid. I've been
>>> > trying to wrap my head around the DM and DMDA parts of PETSc, and as
>>> far
>>> > as I understand, these will help me solve the pressure Poisson equation
>>> > on a decomposed domain (and with geometric multigrid via Galerkin)
>>> > fairly easily.
>>> >
>>> > The tricky part, then; it seems that I must handle "the rest" of the
>>> > domain decomposition myself. Omitting some detail, this means my code
>>> will:
>>> >
>>> > * set up parameters, initial conditions, etc.
>>> > * decompose my array for the velocity field into several parts,
>>> > * time loop:
>>> >        * communicate e.g. the velocity field on the boundaries
>>> >        * each mpi worker will calculate on the local domain the
>>> >          intermediate velocity field, the rhs to the Poisson equation
>>> >          and set up the correct sparse matrix
>>> >        * PETSc will solve the Poisson equation to give me the pressure
>>> >        * each mpi worker will then calculate the updated
>>> >          divergence-free velocity field
>>> >        * each mpi worker will calculate the time step (CFL condition),
>>> >          and we choose the lowest dt among all nodes
>>> > * end time loop
>>> >
>>> > Have I misunderstood anything here? At first I thought the DMDA would
>>> > give me the framework for decomposing the velocity field, handling
>>> > communication of the ghost values at the boundaries etc, but it seems
>>> > this is not the case?
>>> >
>>> > One further question: is it a good idea to set up the DMDA letting
>>> PETSc
>>> > decide the number of processors in each direction, and then using this
>>> > same partition for the rest of my code?
>>> >
>>> > If there are any unclear details, please ask. If it matters, I am using
>>> > the level-set and ghost-fluid methods, so the matrix for my Poisson
>>> > equation must be recomputed each time step. I believe this is the same
>>> > situation as Michele Rosso who posted on this list recently.
>>> >
>>> > Best regards,
>>> > ?smund Ervik
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/3ac3f765/attachment.html>

From pengxwang at hotmail.com  Fri Oct  4 15:04:02 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Fri, 4 Oct 2013 15:04:02 -0500
Subject: [petsc-users] cannot find PetscViewerHDF5 functions in libraries
In-Reply-To: <CAMYG4G=zWfY3wC_MjxUZfus18wRhAK8EyX19=P8bCwV7vKfbPg@mail.gmail.com>
References: <BAY179-W5063FBF92A7DF84722E739B4100@phx.gbl>,
	<CAMYG4G=zWfY3wC_MjxUZfus18wRhAK8EyX19=P8bCwV7vKfbPg@mail.gmail.com>
Message-ID: <BAY179-W823D6F33D15029BF00F9E4B4100@phx.gbl>


Thanks a lot. It works.
Date: Fri, 4 Oct 2013 14:00:09 -0500
Subject: Re: [petsc-users] cannot find PetscViewerHDF5 functions in libraries
From: knepley at gmail.com
To: pengxwang at hotmail.com
CC: petsc-users at mcs.anl.gov

On Fri, Oct 4, 2013 at 1:48 PM, Roc Wang <pengxwang at hotmail.com> wrote:




Hello,

   The version of Petsc I am using is 3.3-p6.  I have hdf5 1.8.11 installed as well as. I am trying to use PetscViewerHDF5Open(), PetscViewerHDF5PushGroup(). But the compiling error shows undefined reference to `PetscViewerHDF5Open' and  undefined reference to `PetscViewerHDF5PushGroup'.  I also tried to compile /petsc-3.3-p6/src/vec/vec/examples/tutorials/ex19.c with petsc style makefile, the same error occurs.


 My config options are --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --download-mpich --with-cxx=g++ --download-hypre=1 --download-ml=1. Should I re-install petsc with --download-hdf5=1? Thanks.
 		 	   		  


If we do not install HDF5 for you, you still have to activate it during configure, --with-hdf5-include= --with-hdf5-lib=.
   Matt

-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/360bb713/attachment.html>

From jedbrown at mcs.anl.gov  Fri Oct  4 15:40:28 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 04 Oct 2013 15:40:28 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
Message-ID: <87d2nkzs5f.fsf@mcs.anl.gov>

Matthew Knepley <knepley at gmail.com> writes:

> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
>
>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
>> hdf5 in the background.
>>
>
> I guess not. It was just easier for me to get Xdmf going. We have trial
> CGNS as well.

I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf gives
you a little more flexibility in choosing a schema for the HDF5 file,
but you can put other stuff in CGNS files and Xdmf has its share of
shortcomings.  If you are happy with CGNS, I would say to use it and
report any issues you have.  We'll enhance the implementation so it can
do everything you need.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/5f27f461/attachment.pgp>

From jedbrown at mcs.anl.gov  Fri Oct  4 16:23:00 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 04 Oct 2013 16:23:00 -0500
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <14240863.L6qv0C0M0G@dhcp071>
References: <2569883.TWAWmgZMUo@rigel> <13652175.rAY5a1HGtb@rigel>
	<8738q75emc.fsf@mcs.anl.gov> <14240863.L6qv0C0M0G@dhcp071>
Message-ID: <87siwgybm3.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:

> On Sunday 18 August 2013 08:10:19 Jed Brown wrote:
>> Output uses a collective write, so the granularity of the IO node is
>> probably more relevant for writing (e.g., BG/Q would have one IO node
>> per 128 compute nodes), but almost any chunk size should perform
>> similarly.  It would make a lot more difference for something like
>
> I ran into this on a Cray XK30 and it's certainly not the case there that any 
> chunk size performs even close to similarly: I can get IO throughput from 
> roughly 50 MB/s to 16 GB/s depending on the chunk sizes and number of ranks 
> participating in the MPI IO operation (underneath H5Dwrite()). 

What range of chunk sizes are you using?  For each fixed number of
ranks, how does the performance vary when varying chunk size from, say,
5MB to 500MB?

> Yes, this certainly needs to be considered, too. I guess huge chunks are bad 
> here?

Likely, but depends what you are looking at.

>> Chunk size needs to be collective.  We could compute an average size
>> From each subdomain, but can't just use the subdomain size.
>
> Why not use the size of the local part of the DA/Vec? That would guarantee

That's fine, but the chunk size needs to be *collective* so we need to
do a reduction or otherwise compute the "average size".

>> I think the chunk size (or maximum chunk size) should be settable by the
>> user.
>
> I agree, that would be the best solution.
>
> Is the granularity (number of ranks actually doing disc IO) settable on HDF5 
> side or does that need to be set in MPI-IO?

I'm not sure what you mean.  On a system like BG, the computed nodes are
not connected to disks and instead have to send the data to IO nodes.
The distribution of IO nodes is part of the machine design.  The ranks
participating in IO are just rearranging data before sending it to the
IO nodes.

> Any idea which version of PETSc this fix might get into? I currently keep my 
> own patched version of gr2.c around, which uses local-Vec-size chunks and it 
> works ok, but I'd like to be able to use vanilla PETSc again.

Send a patch (or submit a pull request) against 'maint' and we'll
consider it.  As long as the change doesn't break any existing uses, it
could be merged to 'maint' (thus v3.4.k for k>=3) after testing.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131004/1368a4bb/attachment-0001.pgp>

From balay at mcs.anl.gov  Fri Oct  4 18:11:13 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 4 Oct 2013 18:11:13 -0500 (CDT)
Subject: [petsc-users] Petsc-3.4.2 with MinGW-w64 on Windows 7
In-Reply-To: <CALmcJSB71q8A8i=vyKc5AghVsxBAS9OuNQx0ByHms==DOctUHg@mail.gmail.com>
References: <CALmcJSDfCeTSj9Ct+hk-4NGpAb6DYi-Oz62cFeOjMRA7jLQgvw@mail.gmail.com>
	<alpine.LFD.2.03.1310011651141.1897@mcs.anl.gov>
	<CALmcJSCsvmyUdx_RjdSLqbXt_5D7oosaJk02M=6a_2MOOcsdHw@mail.gmail.com>
	<alpine.LFD.2.03.1310020004130.1897@mcs.anl.gov>
	<CALmcJSB71q8A8i=vyKc5AghVsxBAS9OuNQx0ByHms==DOctUHg@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310041758160.2061@mcs.anl.gov>

On Thu, 3 Oct 2013, Mengda Wu wrote:

> Hi Satis,
> 
>   I've tried to use mingw distributed with cygwin. And it worked well for
> static libraries.
> 
>   Another problem I am facing is I cannot build shared libraries with it
> even I supplied with "--with-shared-libraries=1". Only .a libs are
> generated and there are no dlls.

Do you need dlls? Its possible [with some hacks] - but the fortran
side won't work.

> There was a message saying "CMake setup
> incomplete (status 256), falling back to legacy build" when configuring.

Hm - thats wierd.. cmake should generally work..

> I did install Cygwin's cmake. I don't know if cmake will help for shared
> libraries.

The general guidelines that can work - is:

- have cygwin cmake
- use the following patch for buildsystem to enable a flag for dll/sharedlibs
https://bitbucket.org/petsc/petsc/commits/50bcd979838a41ef30e4433272ba6c2e94f335bb
- configure  with --with-fc=0
- buit libraries making sure cmake is used.

Satish


> 
> Thanks,
> Mengda
> 
> 
> On Wed, Oct 2, 2013 at 1:07 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Tue, 1 Oct 2013, Mengda Wu wrote:
> >
> > > Hi Satish,
> > >
> > >    Thanks a lot for your reply!
> > >
> > >    Is there a way to use the mingw64 which is not shipped with cygwin?
> > For
> > > example, use windows native paths when testing the compiler.
> >
> > Nope - I don't think it will work. Our build tools are primarily work with
> > cygwin paths.
> >
> > Alternative is to use MS/Intel compilers - and we support them with
> > win32fe compiler wrapper.
> >
> > What problem are you having with using mingw distributed with cygwin?
> >
> > Satish
> >
> 


From sonyablade2010 at hotmail.com  Fri Oct  4 23:32:23 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Sat, 5 Oct 2013 05:32:23 +0100
Subject: [petsc-users] Installation Error
Message-ID: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>

Dear All,

I'm receiving the following errors when I'm trying to install the?
the slepc library, I'm attaching the following error messages and content?
of the configure.log file residing in slepc-3.4.2\arch-mswin-c-debug\conf folder?

?
Here is the output from Cygwin terminal?
noname at noname /cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2
$ ./configure
Checking environment...
Checking PETSc installation...
ERROR: Unable to link with PETSc
ERROR: See "arch-mswin-c-debug/conf/configure.log" file for details


HERE IS THE CONTENT OF CONFIGURE.LOG FILE?
================================================================================
Starting Configure Run at Sat Oct ?5 06:59:54 2013
Configure Options: ./configure
Working directory: /cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2
Python version:
2.7.5 (default, Aug 19 2013, 23:22:02)?
[GCC 4.8.1]
make: /usr/bin/make
PETSc source directory: /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2
PETSc install directory: /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug
PETSc version: 3.4.2
PETSc architecture: arch-mswin-c-debug
SLEPc source directory: /cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2
SLEPc install directory: /cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2/arch-mswin-c-debug
SLEPc version: 3.4.2
================================================================================
Checking PETSc installation...
#include "petscksp.h"
int main() {
Vec v; Mat m; KSP k;
PetscInitializeNoArguments();
VecCreate(PETSC_COMM_WORLD,&v);
MatCreate(PETSC_COMM_WORLD,&m);
KSPCreate(PETSC_COMM_WORLD,&k);
return 0;
}
make: Entering directory `/tmp/slepc-0HKYpc'
gcc -o checklink.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -fno-inline -O0 ? -I/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/include -I/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/include -I/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/include/mpiuni ? ?-D__INSDIR__=./ checklink.c
gcc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -fno-inline -O0 ?-o checklink checklink.o ?-L/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib ?-lpetsc -Wl,-rpath,/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib -lflapack -lfblas -lX11 -lpthread -L/usr/lib/gcc/x86_64-pc-cygwin/4.8.1 -L/usr/x86_64-pc-cygwin/lib -lgfortran -lgfortran -lquadmath -lm -lstdc++ -lgdi32 -luser32 -ladvapi32 -lkernel32 -ldl -lgcc_s -ladvapi32 -lshell32 -luser32 -ldl?
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib/libpetsc.a(fp.c.o):fp.c:(.rdata$.refptr._fe_dfl_env[.refptr._fe_dfl_env]+0x0): undefined reference to `_fe_dfl_env'
collect2: error: ld returned 1 exit status
makefile:2: recipe for target `checklink' failed
make: *** [checklink] Error 1
make: Leaving directory `/tmp/slepc-0HKYpc'
ERROR: Unable to link with PETSc

YOUR HELP WILL BE APPRECIATED, 		 	   		  
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: configure.log
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/dd5af3a2/attachment.ksh>

From balay at mcs.anl.gov  Fri Oct  4 23:48:10 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 4 Oct 2013 23:48:10 -0500 (CDT)
Subject: [petsc-users] Installation Error
In-Reply-To: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
Message-ID: <alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>

On Sat, 5 Oct 2013, Sonya Blade wrote:

> Dear All,
> 
> I'm receiving the following errors when I'm trying to install the?
> the slepc library, I'm attaching the following error messages and content?
> of the configure.log file residing in slepc-3.4.2\arch-mswin-c-debug\conf folder?
> 
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib/libpetsc.a(fp.c.o):fp.c:(.rdata$.refptr._fe_dfl_env[.refptr._fe_dfl_env]+0x0): undefined reference to `_fe_dfl_env'

You'll need the following patch to get petsc working with cygwin64/gcc [configure and build with this patch]

https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207

Sr you can edit PETSC_ARCH/include/petscconf.h and remove the flag PETSC_HAVE_FENV_H and rebuild
the libraries.

Satish

From sonyablade2010 at hotmail.com  Sat Oct  5 01:07:59 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Sat, 5 Oct 2013 07:07:59 +0100
Subject: [petsc-users] Installation Error
In-Reply-To: <alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>,
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
Message-ID: <DUB119-W477957D10254798F6F31FAC8110@phx.gbl>

----------------------------------------
> Date: Fri, 4 Oct 2013 23:48:10 -0500
> From: balay at mcs.anl.gov
> To: sonyablade2010 at hotmail.com
> CC: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Installation Error
>
> On Sat, 5 Oct 2013, Sonya Blade wrote:
>
>> Dear All,
>>
>> I'm receiving the following errors when I'm trying to install the
>> the slepc library, I'm attaching the following error messages and content
>> of the configure.log file residing in slepc-3.4.2\arch-mswin-c-debug\conf folder
>>
>> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib/libpetsc.a(fp.c.o):fp.c:(.rdata$.refptr._fe_dfl_env[.refptr._fe_dfl_env]+0x0): undefined reference to `_fe_dfl_env'
>
> You'll need the following patch to get petsc working with cygwin64/gcc [configure and build with this patch]
>
> https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207
>
> Sr you can edit PETSC_ARCH/include/petscconf.h and remove the flag PETSC_HAVE_FENV_H and rebuild
> the libraries.
>
> Satish

Thank you, I'll try this?

Instead of intending only Win64 platform, how can I make the compilation for Petsc and Slepc?
compatible for both platforms Win32 and Win64 with only single compilation, is it possible at all?

REgards, 		 	   		  

From juhaj at iki.fi  Sat Oct  5 02:50:14 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Sat, 05 Oct 2013 09:50:14 +0200
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <87siwgybm3.fsf@mcs.anl.gov>
References: <2569883.TWAWmgZMUo@rigel> <14240863.L6qv0C0M0G@dhcp071>
	<87siwgybm3.fsf@mcs.anl.gov>
Message-ID: <2211556.92L5HQCr7Q@rigel>

> What range of chunk sizes are you using?  For each fixed number of
> ranks, how does the performance vary when varying chunk size from, say,
> 5MB to 500MB?

I didn't write down the results as they were just a byproduct of getting 
usable performance, but I will do that when I have the time.

> > Why not use the size of the local part of the DA/Vec? That would guarantee
> That's fine, but the chunk size needs to be *collective* so we need to
> do a reduction or otherwise compute the "average size".

I guess I was lucky to have every rank's local Vec precisely the same size (no 
wonder: my test was a 1024^3 lattice with 4096 ranks probably means there were 
64^3 each.

What happens when, say, one has three ranks and Vec lengths 120, 120, and 128 
bytes (or 15, 15, and 16 doubles) on the three ranks: the average becomes 
122.667 which isn't even an integer. How should the chunk dimensions look like 
here? Ceil(122.667), perhaps? But then the sum of chunk sizes > data size, so 
there will be extra data in the file (which I presume HDF5 will take care of 
when accessing the file), is that right?

I'm just asking all these stupid questions since you asked for a patch. ;)

> > Is the granularity (number of ranks actually doing disc IO) settable on
> > HDF5 side or does that need to be set in MPI-IO?
> I'm not sure what you mean.  On a system like BG, the computed nodes are
> not connected to disks and instead have to send the data to IO nodes.
> The distribution of IO nodes is part of the machine design.  The ranks
> participating in IO are just rearranging data before sending it to the
> IO nodes.

Sorry, I forgot about BG and its kin. I meant the ranks participating in IO. 
Obviously the number of IO nodes is determined by the hardware. I just had a 
Cray XK30 in my mind, where those ranks participating in IO are the IO nodes, 
too, so I didn't think of making the distinction between "ranks doing MPI-IO" 
and "IO nodes", which I of course should have.

> Send a patch (or submit a pull request) against 'maint' and we'll
> consider it.  As long as the change doesn't break any existing uses, it
> could be merged to 'maint' (thus v3.4.k for k>=3) after testing.

I'll try to get something useful in. What's the timetable?

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------


From knepley at gmail.com  Sat Oct  5 04:57:54 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 5 Oct 2013 04:57:54 -0500
Subject: [petsc-users] Installation Error
In-Reply-To: <DUB119-W477957D10254798F6F31FAC8110@phx.gbl>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
	<DUB119-W477957D10254798F6F31FAC8110@phx.gbl>
Message-ID: <CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>

On Sat, Oct 5, 2013 at 1:07 AM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> ----------------------------------------
> > Date: Fri, 4 Oct 2013 23:48:10 -0500
> > From: balay at mcs.anl.gov
> > To: sonyablade2010 at hotmail.com
> > CC: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Installation Error
> >
> > On Sat, 5 Oct 2013, Sonya Blade wrote:
> >
> >> Dear All,
> >>
> >> I'm receiving the following errors when I'm trying to install the
> >> the slepc library, I'm attaching the following error messages and
> content
> >> of the configure.log file residing in
> slepc-3.4.2\arch-mswin-c-debug\conf folder
> >>
> >>
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib/libpetsc.a(fp.c.o):fp.c:(.rdata$.refptr._fe_dfl_env[.refptr._fe_dfl_env]+0x0):
> undefined reference to `_fe_dfl_env'
> >
> > You'll need the following patch to get petsc working with cygwin64/gcc
> [configure and build with this patch]
> >
> >
> https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207
> >
> > Sr you can edit PETSC_ARCH/include/petscconf.h and remove the flag
> PETSC_HAVE_FENV_H and rebuild
> > the libraries.
> >
> > Satish
>
> Thank you, I'll try this
>
> Instead of intending only Win64 platform, how can I make the compilation
> for Petsc and Slepc
> compatible for both platforms Win32 and Win64 with only single
> compilation, is it possible at all?
>

That will happen if you patch your sources/build with the latest 'maint'
branch of PETSc.

  Thanks,

     Matt


> REgards,




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/3fb0c29f/attachment.html>

From balay at mcs.anl.gov  Sat Oct  5 09:29:40 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Sat, 5 Oct 2013 09:29:40 -0500 (CDT)
Subject: [petsc-users] Installation Error
In-Reply-To: <CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
	<DUB119-W477957D10254798F6F31FAC8110@phx.gbl>
	<CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310050928090.2061@mcs.anl.gov>

On Sat, 5 Oct 2013, Matthew Knepley wrote:

> On Sat, Oct 5, 2013 at 1:07 AM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:
> 
> > ----------------------------------------
> > > Date: Fri, 4 Oct 2013 23:48:10 -0500
> > > From: balay at mcs.anl.gov
> > > To: sonyablade2010 at hotmail.com
> > > CC: petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] Installation Error
> > >
> > > On Sat, 5 Oct 2013, Sonya Blade wrote:
> > >
> > >> Dear All,
> > >>
> > >> I'm receiving the following errors when I'm trying to install the
> > >> the slepc library, I'm attaching the following error messages and
> > content
> > >> of the configure.log file residing in
> > slepc-3.4.2\arch-mswin-c-debug\conf folder
> > >>
> > >>
> > /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/arch-mswin-c-debug/lib/libpetsc.a(fp.c.o):fp.c:(.rdata$.refptr._fe_dfl_env[.refptr._fe_dfl_env]+0x0):
> > undefined reference to `_fe_dfl_env'
> > >
> > > You'll need the following patch to get petsc working with cygwin64/gcc
> > [configure and build with this patch]
> > >
> > >
> > https://bitbucket.org/petsc/petsc/commits/e4773d96ef73f1e0b315c5b0194cae499d71e207
> > >
> > > Sr you can edit PETSC_ARCH/include/petscconf.h and remove the flag
> > PETSC_HAVE_FENV_H and rebuild
> > > the libraries.
> > >
> > > Satish
> >
> > Thank you, I'll try this
> >
> > Instead of intending only Win64 platform, how can I make the compilation
> > for Petsc and Slepc
> > compatible for both platforms Win32 and Win64 with only single
> > compilation, is it possible at all?
> >
> 
> That will happen if you patch your sources/build with the latest 'maint'
> branch of PETSc.

For source comatibility - yes - the above patch - or using 'maint' branch from git will work.

For binary compatibility - if you have 32bit binaries - it should probably run on both win32 and
win64 machines..

Satish

From jedbrown at mcs.anl.gov  Sat Oct  5 13:34:36 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 05 Oct 2013 13:34:36 -0500
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <2211556.92L5HQCr7Q@rigel>
References: <2569883.TWAWmgZMUo@rigel> <14240863.L6qv0C0M0G@dhcp071>
	<87siwgybm3.fsf@mcs.anl.gov> <2211556.92L5HQCr7Q@rigel>
Message-ID: <871u3zy3b7.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:
> What happens when, say, one has three ranks and Vec lengths 120, 120, and 128 
> bytes (or 15, 15, and 16 doubles) on the three ranks: the average becomes 
> 122.667 which isn't even an integer. How should the chunk dimensions look like 
> here? Ceil(122.667), perhaps? But then the sum of chunk sizes > data size, so 
> there will be extra data in the file (which I presume HDF5 will take care of 
> when accessing the file), is that right?

Yes, there will be some extra space, but HDF5 will deal with it.

> I'll try to get something useful in. What's the timetable?

Thanks.  You're the one that asked to use chunking so it's on your time.
If other people ask before you put something together, we'll check in
with you and decide what to do.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/9b70fae8/attachment.pgp>

From juhaj at iki.fi  Sat Oct  5 15:07:38 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Sat, 05 Oct 2013 22:07:38 +0200
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <871u3zy3b7.fsf@mcs.anl.gov>
References: <2569883.TWAWmgZMUo@rigel> <2211556.92L5HQCr7Q@rigel>
	<871u3zy3b7.fsf@mcs.anl.gov>
Message-ID: <1875290.ey94Ks6JiP@rigel>

> Yes, there will be some extra space, but HDF5 will deal with it.

Splendid.

> > I'll try to get something useful in. What's the timetable?
> Thanks.  You're the one that asked to use chunking so it's on your time.

Actually, I didn't ask for it. I only asked for a bug to be fixed. A bug which 
means > 4 GB Vec cannot be saved into a HDF5 file using PETSc VecView, because 
chunking *was* introduced, but with insane chunk sizes (i.e. the *global* 
instead of local Vec size for each chunk).

Anyway, that's just so the record gets set straight: this is NOT a feature 
request, this is a bug.

> If other people ask before you put something together, we'll check in
> with you and decide what to do.

Thanks, but no need to: if I'm too slow, I'm happy with someone else's fix. In 
fact, I do not have much time to use on this, so I really hope someone else 
beats me to it.

But what I really meant by timetable was that when is the next 3.4.x due and 
when do you need the fix if it is to go to the next release?

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------


From rui.silva at uam.es  Wed Oct  2 06:27:50 2013
From: rui.silva at uam.es (Rui Emanuel Ferreira da Silva)
Date: Wed, 02 Oct 2013 13:27:50 +0200
Subject: [petsc-users] Get a smaller vector with PETSc
Message-ID: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>

I am writing to you to ask some technical problems that I am dealing
with the use of PETSc.

The problem that I am facing is the following: I have defined a vector  
in Petsc which was created with VecCreateMPI. After that, I read this  
vector from an external file with a PETSc viewer.

So if the initial vector is V with size N, I want to obtain a smaller  
vector U with size M, where M<N. I have an array of integers that are  
the indexes of V that I want to pass to U.

How can I do this?

Regards

-------------------
Rui Silva
EMTCCM (European Master in Theoretical Chemistry and Computational Modelling)
UAM, Departamento de Qu?mica, M?dulo 13
CAMPUS http://www.uam.es/departamentos/ciencias/quimica/spline/index.html
-------------------


From rui.silva at uam.es  Wed Oct  2 07:26:08 2013
From: rui.silva at uam.es (Rui Emanuel Ferreira da Silva)
Date: Wed, 02 Oct 2013 14:26:08 +0200
Subject: [petsc-users] Subvector on PETSc
Message-ID: <20131002142608.Horde.RleVR_JsT11STBDgj1sAX2A@webmail.uam.es>

Hello

I am writing to you to ask some technical problems that I am dealing
with the use of PETSc.

The problem that I am facing is the following: I have defined a vector
in Petsc which was created with VecCreateMPI. After that, I read this
vector from an external file with a PETSc viewer.

So if the initial vector is V with size N, I want to obtain a smaller
vector U with size M, where M<N. I have an array of integers that are
the indexes of V that I want to pass to U.

How can I do this?

Regards



-------------------
Rui Silva
EMTCCM (European Master in Theoretical Chemistry and Computational Modelling)
UAM, Departamento de Qu?mica, M?dulo 13
CAMPUS http://www.uam.es/departamentos/ciencias/quimica/spline/index.html
-------------------


From bsmith at mcs.anl.gov  Sat Oct  5 20:59:41 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 5 Oct 2013 20:59:41 -0500
Subject: [petsc-users] Get a smaller vector with PETSc
In-Reply-To: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>
References: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>
Message-ID: <1576B5B5-73F9-4604-9708-CC3634005607@mcs.anl.gov>


  VecScatterCreate().

   If you wish the sub vector to live on a single process then create a sequential vector of the appropriate size on that one process and sequential vectors of size zero on all other processes and scatter into them.

   Barry

On Oct 2, 2013, at 6:27 AM, Rui Emanuel Ferreira da Silva <rui.silva at uam.es> wrote:

> I am writing to you to ask some technical problems that I am dealing
> with the use of PETSc.
> 
> The problem that I am facing is the following: I have defined a vector in Petsc which was created with VecCreateMPI. After that, I read this vector from an external file with a PETSc viewer.
> 
> So if the initial vector is V with size N, I want to obtain a smaller vector U with size M, where M<N. I have an array of integers that are the indexes of V that I want to pass to U.
> 
> How can I do this?
> 
> Regards
> 
> -------------------
> Rui Silva
> EMTCCM (European Master in Theoretical Chemistry and Computational Modelling)
> UAM, Departamento de Qu?mica, M?dulo 13
> CAMPUS http://www.uam.es/departamentos/ciencias/quimica/spline/index.html
> -------------------
> 


From jbermeol at purdue.edu  Sat Oct  5 22:49:41 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sat, 5 Oct 2013 23:49:41 -0400 (EDT)
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <1698666150.232021.1381030464181.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <2047877507.232033.1381031381975.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening.

Thanks

mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed
	Nonzeros in L       10
	Nonzeros in U       10
	nonzeros in L+U     10
	nonzeros in LSUB    10
	NUMfact space (MB) sum(procs):  L\U	0.00	all	0.03
	Total highmark (MB):  All	0.03	Avg	0.02	Max	0.02
        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
                              0.000146866 / 0.000145912 / 0.000146389
	EQUIL time             0.00
	ROWPERM time           0.00
	COLPERM time           0.00
	SYMBFACT time          0.00
	DISTRIBUTE time        0.00
	FACTOR time            0.00
	Factor flops	1.000000e+02	Mflops 	    0.31
	SOLVE time             0.00
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: [1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
Signal received!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct  5 23:45:21 2013
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
[1]PETSC ERROR: Configure run at Sat Oct  5 11:19:36 2013
[1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing
Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct  5 23:45:21 2013
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
[0]PETSC ERROR: Configure run at Sat Oct  5 11:19:36 2013
[0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_superlu_dist.c
Type: text/x-csrc
Size: 4449 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/d1fd93ef/attachment-0001.c>

From jedbrown at mcs.anl.gov  Sat Oct  5 22:54:36 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 05 Oct 2013 22:54:36 -0500
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <1875290.ey94Ks6JiP@rigel>
References: <2569883.TWAWmgZMUo@rigel> <2211556.92L5HQCr7Q@rigel>
	<871u3zy3b7.fsf@mcs.anl.gov> <1875290.ey94Ks6JiP@rigel>
Message-ID: <87a9invytf.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:

> Actually, I didn't ask for it. I only asked for a bug to be fixed. A bug which 
> means > 4 GB Vec cannot be saved into a HDF5 file using PETSc VecView, because 
> chunking *was* introduced, but with insane chunk sizes (i.e. the *global* 
> instead of local Vec size for each chunk).

Ah, right.  I would note that using the local size is also flawed
because we can only have 65k chunks, but we sometimes run jobs with more
than that number of processes.  Maybe we need something like this?

  chunk_size = min(vec_size, max(avg_local_vec_size, vec_size/65k, 10 MiB), 4 GiB)

The combination of the number of chunks (65k) and chunk size (4GiB)
makes for messy constraints.

> But what I really meant by timetable was that when is the next 3.4.x due and 
> when do you need the fix if it is to go to the next release?

I think we're planning to tag 3.4.3 in the next couple weeks.  There
might be a 3.4.4 as well, but I could see going straight to 3.5.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/553908e6/attachment.pgp>

From knepley at gmail.com  Sat Oct  5 22:55:24 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 5 Oct 2013 22:55:24 -0500
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <2047877507.232033.1381031381975.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1698666150.232021.1381030464181.JavaMail.root@mailhub059.itcs.purdue.edu>
	<2047877507.232033.1381031381975.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4G=6LQp3tGX=AiwhQ405N5PoOtHeOq9d3qDd5Jg4mQGw4A@mail.gmail.com>

On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> Hi I'm runnig petsc trying to solve a linear system with superlu_dist.
> However i have a memory violation, atached is the code, and here is the
> output. Email me if you need something else to figured out what is
> happening.
>

So it looks like SuperLU_Dist is bombing during an LAPACK operation. It
could be an MKL problem, or a SuperLU_Dist problem, or our problem,
or a mismatch between versions. I would try to simplify the configuration
in order to cut down on the possibilities. Eliminate everything that is not
necessary for SuperLU_dist first. Then change to --download-f-blas-lapack.
If you still have a crash, send us the matrix since that should be
reproducible and we can report a SuperLU_dist bug or fix our code.

  Thanks,

      Matt


> Thanks
>
> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint
> -mat_superlu_dist_matinput distributed
>         Nonzeros in L       10
>         Nonzeros in U       10
>         nonzeros in L+U     10
>         nonzeros in LSUB    10
>         NUMfact space (MB) sum(procs):  L\U     0.00    all     0.03
>         Total highmark (MB):  All       0.03    Avg     0.02    Max
> 0.02
>         Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg):
>                               0.000146866 / 0.000145912 / 0.000146389
>         EQUIL time             0.00
>         ROWPERM time           0.00
>         COLPERM time           0.00
>         SYMBFACT time          0.00
>         DISTRIBUTE time        0.00
>         FACTOR time            0.00
>         Factor flops    1.000000e+02    Mflops      0.31
>         SOLVE time             0.00
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: [1]PETSC ERROR:       INSTEAD the line number of the start
> of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [1] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> Signal received!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct  5 23:45:21 2013
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [1]PETSC ERROR: Configure run at Sat Oct  5 11:19:36 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
> --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1
> --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1
> --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct  5 23:45:21 2013
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [0]PETSC ERROR: Configure run at Sat Oct  5 11:19:36 2013
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process
> 1
> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
> --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1
> --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131005/56d26ec6/attachment.html>

From jbermeol at purdue.edu  Sun Oct  6 00:10:07 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 01:10:07 -0400 (EDT)
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <CAMYG4G=6LQp3tGX=AiwhQ405N5PoOtHeOq9d3qDd5Jg4mQGw4A@mail.gmail.com>
Message-ID: <1617526363.232103.1381036207524.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi again, I compile with the following configuration:
     --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis

And my code run perfec, so that means is a MKL problem or a mismatch between versions, so how to test what is the problem??

Thanks

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>, petsc-users at mcs.anl.gov
Sent: Saturday, October 5, 2013 11:55:24 PM
Subject: Re: [petsc-users] Superlu_dist error


On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 




Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening. 



So it looks like SuperLU_Dist is bombing during an LAPACK operation. It could be an MKL problem, or a SuperLU_Dist problem, or our problem, 
or a mismatch between versions. I would try to simplify the configuration in order to cut down on the possibilities. Eliminate everything that is not 
necessary for SuperLU_dist first. Then change to --download-f-blas-lapack. If you still have a crash, send us the matrix since that should be 
reproducible and we can report a SuperLU_dist bug or fix our code. 


Thanks, 


Matt 


Thanks 

mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed 
Nonzeros in L 10 
Nonzeros in U 10 
nonzeros in L+U 10 
nonzeros in LSUB 10 
NUMfact space (MB) sum(procs): L\U 0.00 all 0.03 
Total highmark (MB): All 0.03 Avg 0.02 Max 0.02 
Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
0.000146866 / 0.000145912 / 0.000146389 
EQUIL time 0.00 
ROWPERM time 0.00 
COLPERM time 0.00 
SYMBFACT time 0.00 
DISTRIBUTE time 0.00 
FACTOR time 0.00 
Factor flops 1.000000e+02 Mflops 0.31 
SOLVE time 0.00 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger 
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
Try option -start_in_debugger or -on_error_attach_debugger 
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
[0]PETSC ERROR: likely location of problem given in stack below 
[1]PETSC ERROR: likely location of problem given in stack below 
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
[0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of the function 
[1]PETSC ERROR: is given. 
[1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[1]PETSC ERROR: INSTEAD the line number of the start of the function 
[0]PETSC ERROR: is given. 
[0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
[1]PETSC ERROR: --------------------- Error Message ------------------------------------ 
[1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
Signal received! 
[1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[1]PETSC ERROR: See docs/changes/index.html for recent updates. 
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. 
[1]PETSC ERROR: See docs/index.html for manual pages. 
[1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
[1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
[1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
Signal received! 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates. 
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
See docs/faq.html for hints about trouble shooting. 
[0]PETSC ERROR: See docs/index.html for manual pages. 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
[0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
[0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 
Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 



-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener

From sonyablade2010 at hotmail.com  Sun Oct  6 03:03:25 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Sun, 6 Oct 2013 09:03:25 +0100
Subject: [petsc-users] Installation Error
In-Reply-To: <alpine.LFD.2.03.1310050928090.2061@mcs.anl.gov>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
	<DUB119-W477957D10254798F6F31FAC8110@phx.gbl>
	<CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>,
	<alpine.LFD.2.03.1310050928090.2061@mcs.anl.gov>
Message-ID: <DUB119-W2AB34817078825E15C781C8120@phx.gbl>

Thank you all,

I gave up using the cygwin64, in any case compiling with classic cygwin_win32 must work in?
all platforms and is much more convenient for me.?

After succesfully installing petsc and slepc (ver 3.4.2 ) I get the following errors. I'm using?
Code Blocks as IDE editor and I set and configured all the required search directories, external libraries
e.g. lipetsc.a, libslepc.a etc.. in C::B But again compiler raises the following errors. ?

I don't think that its related with Petsc or Slepc installation, it seems that those intrinsically?
or inherently belong to the compiler misconfiguration, but mine is just a guess.

I also would like to note that I'm using GNU GCC Compiler in C:B for compilation, or should I use CYGWin GCC?

Your advice will be appreciated,


.\..\..\slepc-3.4.2\arch-mswin-c-debug\lib\libslepc.a(slepcinit.c.o): In function `SlepcInitialize':
/cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2/src/sys/slepcinit.c:200: undefined reference to `srand48'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o): In function `PetscInitialize':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:659: undefined reference to `__getreent'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:660: undefined reference to `__getreent'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o): In function `PetscFinalize':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1028: undefined reference to `__getreent'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1098: undefined reference to `__getreent'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1162: undefined reference to `__getreent'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o):/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1189: more undefined references to `__getreent' follow
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(mprint.c.o): In function `PetscSynchronizedFGets':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/fileio/mprint.c:663: undefined reference to `__errno'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(checkptr.c.o): In function `PetscCheckPointer':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:29: undefined reference to `sigemptyset'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:32: undefined reference to `sigaction'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:34: undefined reference to `setjmp'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:36: undefined reference to `sigaction'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:74: undefined reference to `sigaction'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(mal.c.o): In function `PetscMallocAlign':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/memory/mal.c:27: undefined reference to `memalign'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(options.c.o): In function `PetscOptionsValidKey':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:242: undefined reference to `_imp____ctype_ptr__'
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:243: undefined reference to `_imp____ctype_ptr__'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(options.c.o): In function `PetscOptionsInsert':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:598: undefined reference to `__getreent'
..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(filev.c.o): In function `PetscViewerFileClose_ASCII':
/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/classes/viewer/impls/ascii/filev.c:17: undefined reference to `__getreent' 		 	   		  

From knepley at gmail.com  Sun Oct  6 07:19:30 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 6 Oct 2013 07:19:30 -0500
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <1617526363.232103.1381036207524.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <CAMYG4G=6LQp3tGX=AiwhQ405N5PoOtHeOq9d3qDd5Jg4mQGw4A@mail.gmail.com>
	<1617526363.232103.1381036207524.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4G=U6PWwftoRzE52M34xg4pd0uMnUe9aNxnuEzW8vVFaLg@mail.gmail.com>

On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> Hi again, I compile with the following configuration:
>      --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
> --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1
> --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0
> --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes
> --download-parmetis=yes --download-metis
>

Get rid of:

  --with-pic=1 --with-fortran=1 --with-fortran-kernels=0

since they do not really do anything, and just put back MKL. I suspect you
will get a crash, and then
it sounds like an MKL bug, or a bizarre incompatibility between SuperLU and
MKL. If not, we explore further.

  Thanks,

     Matt


> And my code run perfec, so that means is a MKL problem or a mismatch
> between versions, so how to test what is the problem??
>
> Thanks
>
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>,
> petsc-users at mcs.anl.gov
> Sent: Saturday, October 5, 2013 11:55:24 PM
> Subject: Re: [petsc-users] Superlu_dist error
>
>
> On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu> wrote:
>
>
>
>
> Hi I'm runnig petsc trying to solve a linear system with superlu_dist.
> However i have a memory violation, atached is the code, and here is the
> output. Email me if you need something else to figured out what is
> happening.
>
>
>
> So it looks like SuperLU_Dist is bombing during an LAPACK operation. It
> could be an MKL problem, or a SuperLU_Dist problem, or our problem,
> or a mismatch between versions. I would try to simplify the configuration
> in order to cut down on the possibilities. Eliminate everything that is not
> necessary for SuperLU_dist first. Then change to --download-f-blas-lapack.
> If you still have a crash, send us the matrix since that should be
> reproducible and we can report a SuperLU_dist bug or fix our code.
>
>
> Thanks,
>
>
> Matt
>
>
> Thanks
>
> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint
> -mat_superlu_dist_matinput distributed
> Nonzeros in L 10
> Nonzeros in U 10
> nonzeros in L+U 10
> nonzeros in LSUB 10
> NUMfact space (MB) sum(procs): L\U 0.00 all 0.03
> Total highmark (MB): All 0.03 Avg 0.02 Max 0.02
> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg):
> 0.000146866 / 0.000145912 / 0.000146389
> EQUIL time 0.00
> ROWPERM time 0.00
> COLPERM time 0.00
> SYMBFACT time 0.00
> DISTRIBUTE time 0.00
> FACTOR time 0.00
> Factor flops 1.000000e+02 Mflops 0.31
> SOLVE time 0.00
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of
> the function
> [1]PETSC ERROR: is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: INSTEAD the line number of the start of the function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [1] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> Signal received!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
> --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1
> --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1
> --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process
> 1
> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
> --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1
> --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/7276f26d/attachment.html>

From knepley at gmail.com  Sun Oct  6 07:38:14 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 6 Oct 2013 07:38:14 -0500
Subject: [petsc-users] Installation Error
In-Reply-To: <DUB119-W2AB34817078825E15C781C8120@phx.gbl>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>
	<DUB119-W477957D10254798F6F31FAC8110@phx.gbl>
	<CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>
	<alpine.LFD.2.03.1310050928090.2061@mcs.anl.gov>
	<DUB119-W2AB34817078825E15C781C8120@phx.gbl>
Message-ID: <CAMYG4G=uqsp4ZL1gdDRfuEmWV2t7kjsYDr_51AnBJcUfhH90Xg@mail.gmail.com>

On Sun, Oct 6, 2013 at 3:03 AM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> Thank you all,
>
> I gave up using the cygwin64, in any case compiling with classic
> cygwin_win32 must work in
> all platforms and is much more convenient for me.
>
> After succesfully installing petsc and slepc (ver 3.4.2 ) I get the
> following errors. I'm using
> Code Blocks as IDE editor and I set and configured all the required search
> directories, external libraries
> e.g. lipetsc.a, libslepc.a etc.. in C::B But again compiler raises the
> following errors.
>
> I don't think that its related with Petsc or Slepc installation, it seems
> that those intrinsically
> or inherently belong to the compiler misconfiguration, but mine is just a
> guess.
>
> I also would like to note that I'm using GNU GCC Compiler in C:B for
> compilation, or should I use CYGWin GCC?
>

1) You must use the same compiler that you configured with.

2) You can get all the libraries you need to specify using

   make get linklibs

   Matt


> Your advice will be appreciated,
>
>
> .\..\..\slepc-3.4.2\arch-mswin-c-debug\lib\libslepc.a(slepcinit.c.o): In
> function `SlepcInitialize':
> /cygdrive/d/TEST_FOLDER_asus/slepc-3.4.2/src/sys/slepcinit.c:200:
> undefined reference to `srand48'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o): In
> function `PetscInitialize':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:659:
> undefined reference to `__getreent'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:660:
> undefined reference to `__getreent'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o): In
> function `PetscFinalize':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1028:
> undefined reference to `__getreent'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1098:
> undefined reference to `__getreent'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1162:
> undefined reference to `__getreent'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(pinit.c.o):/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/pinit.c:1189:
> more undefined references to `__getreent' follow
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(mprint.c.o): In
> function `PetscSynchronizedFGets':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/fileio/mprint.c:663:
> undefined reference to `__errno'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(checkptr.c.o):
> In function `PetscCheckPointer':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:29:
> undefined reference to `sigemptyset'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:32:
> undefined reference to `sigaction'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:34:
> undefined reference to `setjmp'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:36:
> undefined reference to `sigaction'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/error/checkptr.c:74:
> undefined reference to `sigaction'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(mal.c.o): In
> function `PetscMallocAlign':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/memory/mal.c:27:
> undefined reference to `memalign'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(options.c.o): In
> function `PetscOptionsValidKey':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:242:
> undefined reference to `_imp____ctype_ptr__'
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:243:
> undefined reference to `_imp____ctype_ptr__'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(options.c.o): In
> function `PetscOptionsInsert':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:598:
> undefined reference to `__getreent'
> ..\..\..\..\petsc-3.4.2\arch-mswin-c-debug\lib\libpetsc.a(filev.c.o): In
> function `PetscViewerFileClose_ASCII':
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/classes/viewer/impls/ascii/filev.c:17:
> undefined reference to `__getreent'




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/78cac663/attachment-0001.html>

From juhaj at iki.fi  Sun Oct  6 11:59:37 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Sun, 06 Oct 2013 18:59:37 +0200
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <87a9invytf.fsf@mcs.anl.gov>
References: <2569883.TWAWmgZMUo@rigel> <1875290.ey94Ks6JiP@rigel>
	<87a9invytf.fsf@mcs.anl.gov>
Message-ID: <7639338.FUb38SSzMH@rigel>

> > Actually, I didn't ask for it. I only asked for a bug to be fixed. A bug
> > which means > 4 GB Vec cannot be saved into a HDF5 file using PETSc
> > VecView, because chunking *was* introduced, but with insane chunk sizes
> Ah, right.  I would note that using the local size is also flawed
> because we can only have 65k chunks, but we sometimes run jobs with more
> than that number of processes.  Maybe we need something like this?
> 
>   chunk_size = min(vec_size, max(avg_local_vec_size, vec_size/65k, 10 MiB),
> 4 GiB)

Argh, messy indeed. Are you sure you mean 65 k and not 64 Ki? I made a small 
table of the situation just to make sure I am not missing anything. In the 
table, "small" means < 4 GB, "large" means >= 4 GB, "few" means < 65 k, "many" 
means >= 65 k. Note that local size > global size is impossible, but I include 
the row on the table for completeness's sake.

Variables: 	local size	global size	# ranks		chunks
		small		small		few		global size
		small		small		many		global size[1]	
		small		large		few		avg local size
		small		large		many		4 GiB
		large		small		few		impossible
		large		small		many		impossible
		large		large		few		4 GiB[2]
		large		large		many		65 k chunks

[1] It sounds improbable anyone would run a problem with < 4 GiB data with >= 
65k ranks, but fortunately it's not a problem.

[2] Unless I'm mistaken, this situation will always give < 65 k chunks for 4 
GiB chunk size.

I also believe your formula gives "the right" answer in each case. Just one 
more question: is "average local size" a good solution or is it better to use 
"max local size"? The latter will cause more unnecessary data in the file, but 
unless I'm mistaken, the former will require extra MPI communication to fill 
in the portions of ranks whose local size is less than average.

HDF5 really needs to fix this internally. As it stands, a single HDF5 dataset 
cannot hold more than 260 TiB ? not that many people would want such files 
anyway, but then again, "640 kiB should be enough for everybody", right? I'm 
running simulations which take more than terabyte of memory, and I'm by far 
not the biggest memory consumer in the world, so the limit is not really as 
far as it might seem.

> I think we're planning to tag 3.4.3 in the next couple weeks.  There
> might be a 3.4.4 as well, but I could see going straight to 3.5.

Ok. I don't see myself having time to fix and test this in two weeks, but 
3.4.4 should be doable. Anyone else want to fix the bug by then?

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------


From jedbrown at mcs.anl.gov  Sun Oct  6 12:24:58 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 06 Oct 2013 12:24:58 -0500
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <7639338.FUb38SSzMH@rigel>
References: <2569883.TWAWmgZMUo@rigel> <1875290.ey94Ks6JiP@rigel>
	<87a9invytf.fsf@mcs.anl.gov> <7639338.FUb38SSzMH@rigel>
Message-ID: <8738oewbv9.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:

> Argh, messy indeed. Are you sure you mean 65 k and not 64 Ki? 

I was using 65 as shorthand for 2^{32}, yes.

> I made a small table of the situation just to make sure I am not
> missing anything. In the table, "small" means < 4 GB, "large" means >=
> 4 GB, "few" means < 65 k, "many" means >= 65 k. Note that local size >
> global size is impossible, but I include the row on the table for
> completeness's sake.
>
> Variables: 	local size	global size	# ranks		chunks
> 		small		small		few		global size
> 		small		small		many		global size[1]	
> 		small		large		few		avg local size
> 		small		large		many		4 GiB
> 		large		small		few		impossible
> 		large		small		many		impossible
> 		large		large		few		4 GiB[2]
> 		large		large		many		65 k chunks

The last line cannot be stored since it consists of more than 65k chunks
of size larger than 4 GiB.

> [1] It sounds improbable anyone would run a problem with < 4 GiB data with >= 
> 65k ranks, but fortunately it's not a problem.

Au contraire, it's typical for strong scaling to roll over at about 1000
dofs/core (about 10 kB of data), so 65k ranks is still less than 1 GB.

> [2] Unless I'm mistaken, this situation will always give < 65 k chunks for 4 
> GiB chunk size.

Well, 33k ranks each trying to write 8 GiB would need 66k chunks, but
there is no way to write this file so we don't need to worry about it.

> I also believe your formula gives "the right" answer in each case. Just one 
> more question: is "average local size" a good solution or is it better to use 
> "max local size"? The latter will cause more unnecessary data in the file, but 
> unless I'm mistaken, the former will require extra MPI communication to fill 
> in the portions of ranks whose local size is less than average.

It depends how the compute nodes are connected to the file system, but
even if you use "average" and if size is statistically uncorrelated with
rank but has positive variance, the expected value of the skew will be
more than a rank's contribution for sufficiently large numbers of cores.
In other words, it's not possible to "balance" without needing to move
all data for a sufficiently large number of ranks.

> HDF5 really needs to fix this internally. As it stands, a single HDF5 dataset 
> cannot hold more than 260 TiB ? not that many people would want such files 
> anyway, but then again, "640 kiB should be enough for everybody", right? I'm 
> running simulations which take more than terabyte of memory, and I'm by far 
> not the biggest memory consumer in the world, so the limit is not really as 
> far as it might seem.

We're already there.  Today's large machines can fit several vectors of
size 260 TB in memory.

>> I think we're planning to tag 3.4.3 in the next couple weeks.  There
>> might be a 3.4.4 as well, but I could see going straight to 3.5.
>
> Ok. I don't see myself having time to fix and test this in two weeks, but 
> 3.4.4 should be doable. Anyone else want to fix the bug by then?

I'll write back if I get to it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/c2dd19ce/attachment.pgp>

From jbermeol at purdue.edu  Sun Oct  6 12:52:00 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 13:52:00 -0400 (EDT)
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <CAMYG4G=U6PWwftoRzE52M34xg4pd0uMnUe9aNxnuEzW8vVFaLg@mail.gmail.com>
Message-ID: <1218310234.232589.1381081920901.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi again, now I'm getting the error again. The problem happens when I'm using the flag -O3 for the compiler. So what should I do next to solve this??

Attached is my code, and for tis example I'm creating the matrix in the code.

Thanks  

Nonzeros in L       10
	Nonzeros in U       10
	nonzeros in L+U     10
	nonzeros in LSUB    10
	NUMfact space (MB) sum(procs):  L\U	0.00	all	0.03
	Total highmark (MB):  All	0.03	Avg	0.02	Max	0.02
        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
                              0.000124216 / 4.81606e-05 / 8.61883e-05
	EQUIL time             0.00
	ROWPERM time           0.00
	COLPERM time           0.00
	SYMBFACT time          0.00
	DISTRIBUTE time        0.00
	FACTOR time            0.00
	Factor flops	1.000000e+02	Mflops 	    0.31
	SOLVE time             0.00
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[1]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Signal received!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
[1]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
[1]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
[1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
[0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
[0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-03 CXXOPTFLAGS=-03 FOPTFLAGS=-03
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>, petsc-users at mcs.anl.gov
Sent: Sunday, October 6, 2013 8:19:30 AM
Subject: Re: [petsc-users] Superlu_dist error


On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 




Hi again, I compile with the following configuration: 
--with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis 



Get rid of: 


--with-pic=1 --with-fortran=1 --with-fortran-kernels=0 


since they do not really do anything, and just put back MKL. I suspect you will get a crash, and then 
it sounds like an MKL bug, or a bizarre incompatibility between SuperLU and MKL. If not, we explore further. 


Thanks, 


Matt 


And my code run perfec, so that means is a MKL problem or a mismatch between versions, so how to test what is the problem?? 

Thanks 

----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "Jose David Bermeol" < jbermeol at purdue.edu > 
Cc: "For users of the development version of PETSc" < petsc-dev at mcs.anl.gov >, petsc-users at mcs.anl.gov 
Sent: Saturday, October 5, 2013 11:55:24 PM 
Subject: Re: [petsc-users] Superlu_dist error 


On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 




Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening. 



So it looks like SuperLU_Dist is bombing during an LAPACK operation. It could be an MKL problem, or a SuperLU_Dist problem, or our problem, 
or a mismatch between versions. I would try to simplify the configuration in order to cut down on the possibilities. Eliminate everything that is not 
necessary for SuperLU_dist first. Then change to --download-f-blas-lapack. If you still have a crash, send us the matrix since that should be 
reproducible and we can report a SuperLU_dist bug or fix our code. 


Thanks, 


Matt 


Thanks 

mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed 
Nonzeros in L 10 
Nonzeros in U 10 
nonzeros in L+U 10 
nonzeros in LSUB 10 
NUMfact space (MB) sum(procs): L\U 0.00 all 0.03 
Total highmark (MB): All 0.03 Avg 0.02 Max 0.02 
Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
0.000146866 / 0.000145912 / 0.000146389 
EQUIL time 0.00 
ROWPERM time 0.00 
COLPERM time 0.00 
SYMBFACT time 0.00 
DISTRIBUTE time 0.00 
FACTOR time 0.00 
Factor flops 1.000000e+02 Mflops 0.31 
SOLVE time 0.00 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger 
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
Try option -start_in_debugger or -on_error_attach_debugger 
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
[0]PETSC ERROR: likely location of problem given in stack below 
[1]PETSC ERROR: likely location of problem given in stack below 
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
[0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of the function 
[1]PETSC ERROR: is given. 
[1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[1]PETSC ERROR: INSTEAD the line number of the start of the function 
[0]PETSC ERROR: is given. 
[0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
[1]PETSC ERROR: --------------------- Error Message ------------------------------------ 
[1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
[0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
Signal received! 
[1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[1]PETSC ERROR: See docs/changes/index.html for recent updates. 
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. 
[1]PETSC ERROR: See docs/index.html for manual pages. 
[1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
[1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
[1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
Signal received! 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates. 
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
[1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
See docs/faq.html for hints about trouble shooting. 
[0]PETSC ERROR: See docs/index.html for manual pages. 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
[0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
[0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
[0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 
Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
[0]PETSC ERROR: ------------------------------------------------------------------------ 
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 



-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_superlu_dist.c
Type: text/x-csrc
Size: 4449 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/ef026012/attachment.c>

From knepley at gmail.com  Sun Oct  6 13:11:47 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 6 Oct 2013 13:11:47 -0500
Subject: [petsc-users] Superlu_dist error
In-Reply-To: <1218310234.232589.1381081920901.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <CAMYG4G=U6PWwftoRzE52M34xg4pd0uMnUe9aNxnuEzW8vVFaLg@mail.gmail.com>
	<1218310234.232589.1381081920901.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4G=HWAiPuxeLO+2r_awX0Ko+tFvL1Y6srw3GD_CrsyU4bw@mail.gmail.com>

On Sun, Oct 6, 2013 at 12:52 PM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> Hi again, now I'm getting the error again. The problem happens when I'm
> using the flag -O3 for the compiler. So what should I do next to solve
> this??
>

This sounds like an Intel compiler bug. We have seen lots of these.
Honestly, for the kinds of operations in
PETSc, you will see no speed improvement over GNU. Do you use something in
your code that runs faster
with Intel? If so, can you upgrade?

  Thanks,

     Matt


> Attached is my code, and for tis example I'm creating the matrix in the
> code.
>
> Thanks
>
> Nonzeros in L       10
>         Nonzeros in U       10
>         nonzeros in L+U     10
>         nonzeros in LSUB    10
>         NUMfact space (MB) sum(procs):  L\U     0.00    all     0.03
>         Total highmark (MB):  All       0.03    Avg     0.02    Max
> 0.02
>         Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg):
>                               0.000124216 / 4.81606e-05 / 8.61883e-05
>         EQUIL time             0.00
>         ROWPERM time           0.00
>         COLPERM time           0.00
>         SYMBFACT time          0.00
>         DISTRIBUTE time        0.00
>         FACTOR time            0.00
>         Factor flops    1.000000e+02    Mflops      0.31
>         SOLVE time             0.00
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve line 3207
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Signal received!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named
> carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [1]PETSC ERROR: Libraries linked from
> /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [1]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
> --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1
> --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes
> --download-superlu=yes --download-parmetis=yes --download-metis
> COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process
> 1
> Caught signal number 11 SEGV: Segmentation Violation, probably memory
> access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207
> /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named
> carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
> --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1
> --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes
> --download-superlu=yes --download-parmetis=yes --download-metis
> COPTFLAGS=-03 CXXOPTFLAGS=-03 FOPTFLAGS=-03
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>,
> petsc-users at mcs.anl.gov
> Sent: Sunday, October 6, 2013 8:19:30 AM
> Subject: Re: [petsc-users] Superlu_dist error
>
>
> On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol < jbermeol at purdue.edu> wrote:
>
>
>
>
> Hi again, I compile with the following configuration:
> --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
> --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1
> --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0
> --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes
> --download-parmetis=yes --download-metis
>
>
>
> Get rid of:
>
>
> --with-pic=1 --with-fortran=1 --with-fortran-kernels=0
>
>
> since they do not really do anything, and just put back MKL. I suspect you
> will get a crash, and then
> it sounds like an MKL bug, or a bizarre incompatibility between SuperLU
> and MKL. If not, we explore further.
>
>
> Thanks,
>
>
> Matt
>
>
> And my code run perfec, so that means is a MKL problem or a mismatch
> between versions, so how to test what is the problem??
>
> Thanks
>
> ----- Original Message -----
> From: "Matthew Knepley" < knepley at gmail.com >
> To: "Jose David Bermeol" < jbermeol at purdue.edu >
> Cc: "For users of the development version of PETSc" <
> petsc-dev at mcs.anl.gov >, petsc-users at mcs.anl.gov
> Sent: Saturday, October 5, 2013 11:55:24 PM
> Subject: Re: [petsc-users] Superlu_dist error
>
>
> On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu> wrote:
>
>
>
>
> Hi I'm runnig petsc trying to solve a linear system with superlu_dist.
> However i have a memory violation, atached is the code, and here is the
> output. Email me if you need something else to figured out what is
> happening.
>
>
>
> So it looks like SuperLU_Dist is bombing during an LAPACK operation. It
> could be an MKL problem, or a SuperLU_Dist problem, or our problem,
> or a mismatch between versions. I would try to simplify the configuration
> in order to cut down on the possibilities. Eliminate everything that is not
> necessary for SuperLU_dist first. Then change to --download-f-blas-lapack.
> If you still have a crash, send us the matrix since that should be
> reproducible and we can report a SuperLU_dist bug or fix our code.
>
>
> Thanks,
>
>
> Matt
>
>
> Thanks
>
> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint
> -mat_superlu_dist_matinput distributed
> Nonzeros in L 10
> Nonzeros in U 10
> nonzeros in L+U 10
> nonzeros in LSUB 10
> NUMfact space (MB) sum(procs): L\U 0.00 all 0.03
> Total highmark (MB): All 0.03 Avg 0.02 Max 0.02
> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg):
> 0.000146866 / 0.000145912 / 0.000146389
> EQUIL time 0.00
> ROWPERM time 0.00
> COLPERM time 0.00
> SYMBFACT time 0.00
> DISTRIBUTE time 0.00
> FACTOR time 0.00
> Factor flops 1.000000e+02 Mflops 0.31
> SOLVE time 0.00
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of
> the function
> [1]PETSC ERROR: is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: INSTEAD the line number of the start of the function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [1] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c
> Signal received!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc
> --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1
> --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1
> --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a linux-complex named
> carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013
> [0]PETSC ERROR: Libraries linked from
> /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib
> [0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process
> 1
> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
> --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1
> --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0
> --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl
> --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so
> --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64
> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"
> --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3
> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3
> --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include
> --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]"
> --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso
> --with-hdf5 --download-hdf5=yes --download-metis=yes
> --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes
> --download-mumps=yes --download-spooles=yes --download-pastix=yes
> --download-ptscotch=yes --download-umfpack=yes --download-sowing
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/0e081974/attachment-0001.html>

From jedbrown at mcs.anl.gov  Sun Oct  6 15:31:18 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 06 Oct 2013 15:31:18 -0500
Subject: [petsc-users] Is Elemental compatible with Clang from LLVM 3.3?
In-Reply-To: <CAHC8rGtFOk4PP_pgwsH6yS8sRPtjC5XEJrKsySGE-fKkeCz=NA@mail.gmail.com>
References: <CAHC8rGtFOk4PP_pgwsH6yS8sRPtjC5XEJrKsySGE-fKkeCz=NA@mail.gmail.com>
Message-ID: <87r4byuoo9.fsf@mcs.anl.gov>

Geoff, thanks for the log.  

| In file included from /Users/goxberry/petsc-3.4.2/src/mat/impls/elemental/matelem.cxx:1:
| In file included from /Users/goxberry/petsc-3.4.2/include/../src/mat/impls/elemental/matelemimpl.h:5:
| In file included from /Users/goxberry/petsc-3.4.2/include/petsc-private/matimpl.h:5:
| In file included from /Users/goxberry/petsc-3.4.2/include/petscmat.h:6:
| In file included from /Users/goxberry/petsc-3.4.2/include/petscvec.h:9:
| In file included from /Users/goxberry/petsc-3.4.2/include/petscis.h:7:
| /Users/goxberry/petsc-3.4.2/include/petscsys.h:2325:117: error: expected ')'
| PETSC_STATIC_INLINE PetscErrorCode PetscSegBufferGetInts(PetscSegBuffer seg,PetscInt count,PetscInt *PETSC_RESTRICT *slot) {return PetscSegBufferGet(seg,count,(void**)slot);}
|                                                                                                                     ^


Karl fixed this in 'next', but in a branch started from 'master'.  We
had a discussion about the design at that point and decided that the
"hack" was messy and error-prone.

https://bitbucket.org/petsc/petsc/commits/0a8720703c3660d0eaeec3a29abfe2a9bfa7773f


I think Matt's suggestion of two separate files is overkill and not
clearly better.  Here is an alternative proposal (which is consistent
with the handling of PETSC_FUNCTION_NAME, which works smoothly).

https://bitbucket.org/petsc/petsc/commits/ed938b00aafbc8619c94f12cc845a1c4a3fa2bcd

You can test by:

  $ git checkout jed/fix-restrict-cxx
  $ ./configure ...

(Actually, you could just delete the definitions of PETSC_RESTRICT and
PETSC_STATIC_INLINE from petscconf.h, then run "make".)

This branch starts from 'maint' so you won't see any interface changes
in the above.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/607bd4a4/attachment.pgp>

From goxberry at mit.edu  Sun Oct  6 14:37:28 2013
From: goxberry at mit.edu (Geoff Oxberry)
Date: Sun, 6 Oct 2013 12:37:28 -0700
Subject: [petsc-users] Is Elemental compatible with Clang from LLVM 3.3?
Message-ID: <CAHC8rGtFOk4PP_pgwsH6yS8sRPtjC5XEJrKsySGE-fKkeCz=NA@mail.gmail.com>

See log files attached. Also, in case it's necessary:

goxberry$ clang --version
Apple LLVM version 5.0 (clang-500.2.75) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix

-- 
Geoffrey Oxberry, Ph.D., E.I.T.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/4b6b47f9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 3726900 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/4b6b47f9/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 257609 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/4b6b47f9/attachment-0003.obj>

From bsmith at mcs.anl.gov  Sun Oct  6 15:44:32 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 6 Oct 2013 15:44:32 -0500
Subject: [petsc-users] [petsc-dev]  Superlu_dist error
In-Reply-To: <1218310234.232589.1381081920901.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1218310234.232589.1381081920901.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <C53AEC14-5A29-425E-9A2C-0586BEEE61C5@mcs.anl.gov>


  If you install with -O instead of -O3 (use a different PETSC_ARCH so it doesn't overwrite) does the problem go away?

   Barry

On Oct 6, 2013, at 12:52 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Hi again, now I'm getting the error again. The problem happens when I'm using the flag -O3 for the compiler. So what should I do next to solve this??
> 
> Attached is my code, and for tis example I'm creating the matrix in the code.
> 
> Thanks  
> 
> Nonzeros in L       10
> 	Nonzeros in U       10
> 	nonzeros in L+U     10
> 	nonzeros in LSUB    10
> 	NUMfact space (MB) sum(procs):  L\U	0.00	all	0.03
> 	Total highmark (MB):  All	0.03	Avg	0.02	Max	0.02
>        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
>                              0.000124216 / 4.81606e-05 / 8.61883e-05
> 	EQUIL time             0.00
> 	ROWPERM time           0.00
> 	COLPERM time           0.00
> 	SYMBFACT time          0.00
> 	DISTRIBUTE time        0.00
> 	FACTOR time            0.00
> 	Factor flops	1.000000e+02	Mflops 	    0.31
> 	SOLVE time             0.00
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Signal received!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [1]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [1]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-03 CXXOPTFLAGS=-03 FOPTFLAGS=-03
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>, petsc-users at mcs.anl.gov
> Sent: Sunday, October 6, 2013 8:19:30 AM
> Subject: Re: [petsc-users] Superlu_dist error
> 
> 
> On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
> 
> 
> 
> 
> Hi again, I compile with the following configuration: 
> --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis 
> 
> 
> 
> Get rid of: 
> 
> 
> --with-pic=1 --with-fortran=1 --with-fortran-kernels=0 
> 
> 
> since they do not really do anything, and just put back MKL. I suspect you will get a crash, and then 
> it sounds like an MKL bug, or a bizarre incompatibility between SuperLU and MKL. If not, we explore further. 
> 
> 
> Thanks, 
> 
> 
> Matt 
> 
> 
> And my code run perfec, so that means is a MKL problem or a mismatch between versions, so how to test what is the problem?? 
> 
> Thanks 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "Jose David Bermeol" < jbermeol at purdue.edu > 
> Cc: "For users of the development version of PETSc" < petsc-dev at mcs.anl.gov >, petsc-users at mcs.anl.gov 
> Sent: Saturday, October 5, 2013 11:55:24 PM 
> Subject: Re: [petsc-users] Superlu_dist error 
> 
> 
> On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
> 
> 
> 
> 
> Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening. 
> 
> 
> 
> So it looks like SuperLU_Dist is bombing during an LAPACK operation. It could be an MKL problem, or a SuperLU_Dist problem, or our problem, 
> or a mismatch between versions. I would try to simplify the configuration in order to cut down on the possibilities. Eliminate everything that is not 
> necessary for SuperLU_dist first. Then change to --download-f-blas-lapack. If you still have a crash, send us the matrix since that should be 
> reproducible and we can report a SuperLU_dist bug or fix our code. 
> 
> 
> Thanks, 
> 
> 
> Matt 
> 
> 
> Thanks 
> 
> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed 
> Nonzeros in L 10 
> Nonzeros in U 10 
> nonzeros in L+U 10 
> nonzeros in LSUB 10 
> NUMfact space (MB) sum(procs): L\U 0.00 all 0.03 
> Total highmark (MB): All 0.03 Avg 0.02 Max 0.02 
> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
> 0.000146866 / 0.000145912 / 0.000146389 
> EQUIL time 0.00 
> ROWPERM time 0.00 
> COLPERM time 0.00 
> SYMBFACT time 0.00 
> DISTRIBUTE time 0.00 
> FACTOR time 0.00 
> Factor flops 1.000000e+02 Mflops 0.31 
> SOLVE time 0.00 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger 
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
> Try option -start_in_debugger or -on_error_attach_debugger 
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
> [0]PETSC ERROR: likely location of problem given in stack below 
> [1]PETSC ERROR: likely location of problem given in stack below 
> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
> [0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of the function 
> [1]PETSC ERROR: is given. 
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [1]PETSC ERROR: INSTEAD the line number of the start of the function 
> [0]PETSC ERROR: is given. 
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ 
> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
> Signal received! 
> [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates. 
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. 
> [1]PETSC ERROR: See docs/index.html for manual pages. 
> [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
> [1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
> Signal received! 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates. 
> [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
> See docs/faq.html for hints about trouble shooting. 
> [0]PETSC ERROR: See docs/index.html for manual pages. 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
> [0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 
> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener<test_superlu_dist.c>


From goxberry at mit.edu  Sun Oct  6 15:59:44 2013
From: goxberry at mit.edu (Geoff Oxberry)
Date: Sun, 6 Oct 2013 13:59:44 -0700
Subject: [petsc-users] Is Elemental compatible with Clang from LLVM 3.3?
In-Reply-To: <87r4byuoo9.fsf@mcs.anl.gov>
References: <CAHC8rGtFOk4PP_pgwsH6yS8sRPtjC5XEJrKsySGE-fKkeCz=NA@mail.gmail.com>
	<87r4byuoo9.fsf@mcs.anl.gov>
Message-ID: <CAHC8rGv5eBQs21B_47Vi+QERer3Z8+wGhUs1R8KT=4-+=niDnw@mail.gmail.com>

Yes, that works. Thanks, Jed!


On Sun, Oct 6, 2013 at 1:31 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Geoff, thanks for the log.
>
> | In file included from
> /Users/goxberry/petsc-3.4.2/src/mat/impls/elemental/matelem.cxx:1:
> | In file included from
> /Users/goxberry/petsc-3.4.2/include/../src/mat/impls/elemental/matelemimpl.h:5:
> | In file included from
> /Users/goxberry/petsc-3.4.2/include/petsc-private/matimpl.h:5:
> | In file included from /Users/goxberry/petsc-3.4.2/include/petscmat.h:6:
> | In file included from /Users/goxberry/petsc-3.4.2/include/petscvec.h:9:
> | In file included from /Users/goxberry/petsc-3.4.2/include/petscis.h:7:
> | /Users/goxberry/petsc-3.4.2/include/petscsys.h:2325:117: error: expected
> ')'
> | PETSC_STATIC_INLINE PetscErrorCode PetscSegBufferGetInts(PetscSegBuffer
> seg,PetscInt count,PetscInt *PETSC_RESTRICT *slot) {return
> PetscSegBufferGet(seg,count,(void**)slot);}
> |
>                                             ^
>
>
> Karl fixed this in 'next', but in a branch started from 'master'.  We
> had a discussion about the design at that point and decided that the
> "hack" was messy and error-prone.
>
>
> https://bitbucket.org/petsc/petsc/commits/0a8720703c3660d0eaeec3a29abfe2a9bfa7773f
>
>
> I think Matt's suggestion of two separate files is overkill and not
> clearly better.  Here is an alternative proposal (which is consistent
> with the handling of PETSC_FUNCTION_NAME, which works smoothly).
>
>
> https://bitbucket.org/petsc/petsc/commits/ed938b00aafbc8619c94f12cc845a1c4a3fa2bcd
>
> You can test by:
>
>   $ git checkout jed/fix-restrict-cxx
>   $ ./configure ...
>
> (Actually, you could just delete the definitions of PETSC_RESTRICT and
> PETSC_STATIC_INLINE from petscconf.h, then run "make".)
>
> This branch starts from 'maint' so you won't see any interface changes
> in the above.
>



-- 
Geoffrey Oxberry, Ph.D., E.I.T.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/18261165/attachment.html>

From jedbrown at mcs.anl.gov  Sun Oct  6 16:24:17 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 06 Oct 2013 16:24:17 -0500
Subject: [petsc-users] Is Elemental compatible with Clang from LLVM 3.3?
In-Reply-To: <CAHC8rGv5eBQs21B_47Vi+QERer3Z8+wGhUs1R8KT=4-+=niDnw@mail.gmail.com>
References: <CAHC8rGtFOk4PP_pgwsH6yS8sRPtjC5XEJrKsySGE-fKkeCz=NA@mail.gmail.com>
	<87r4byuoo9.fsf@mcs.anl.gov>
	<CAHC8rGv5eBQs21B_47Vi+QERer3Z8+wGhUs1R8KT=4-+=niDnw@mail.gmail.com>
Message-ID: <87li26um7y.fsf@mcs.anl.gov>

Geoff Oxberry <goxberry at mit.edu> writes:

> Yes, that works. Thanks, Jed!

Cool, I merged it to 'next' and will get it in v3.4.3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/149d2ab7/attachment.pgp>

From jbermeol at purdue.edu  Sun Oct  6 20:04:32 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 21:04:32 -0400 (EDT)
Subject: [petsc-users] mkl-cpardiso configuration
Message-ID: <2004403497.233618.1381107872511.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:

-Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5

Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great. 

Thanks 

From knepley at gmail.com  Sun Oct  6 20:43:19 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 6 Oct 2013 20:43:19 -0500
Subject: [petsc-users] mkl-cpardiso configuration
In-Reply-To: <2004403497.233618.1381107872511.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <2004403497.233618.1381107872511.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4G=Y__PyzadKOk20fr4LvOrT3K+ivaUrHzUWa3VTfmB8mw@mail.gmail.com>

On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> Hi, one question about how to configure a new package, intel people
> suggest to compile mkl-cpardiso using the following flags for the linker:
>
> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64
> -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
>
> Now I want to be able to do this just using the flag
> --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to
> modify my mkl.py file to achieve this, however your configuration system is
> a little bit complex, so if you code give some guidelines to achieve this
> or files to look how to do it would be great.
>

Is there a reason you do not want to use --with-blas-lapack-lib?

  Thanks,

    Matt


> Thanks
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/a4b22a5f/attachment-0001.html>

From bsmith at mcs.anl.gov  Sun Oct  6 20:49:59 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 6 Oct 2013 20:49:59 -0500
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <CAMYG4G=Y__PyzadKOk20fr4LvOrT3K+ivaUrHzUWa3VTfmB8mw@mail.gmail.com>
References: <2004403497.233618.1381107872511.JavaMail.root@mailhub059.itcs.purdue.edu>
	<CAMYG4G=Y__PyzadKOk20fr4LvOrT3K+ivaUrHzUWa3VTfmB8mw@mail.gmail.com>
Message-ID: <C5A7B0C7-6911-470B-8233-1F5D4BFE7E5D@mcs.anl.gov>


   Because he is using other stuff in mkl besides blas/lapack!

On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
> 
> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
> 
> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.

   Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.

   Barry

> 
> Is there a reason you do not want to use --with-blas-lapack-lib?
> 
>   Thanks,
> 
>     Matt
>  
> Thanks
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From jbermeol at purdue.edu  Sun Oct  6 20:52:07 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 21:52:07 -0400 (EDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <C5A7B0C7-6911-470B-8233-1F5D4BFE7E5D@mcs.anl.gov>
Message-ID: <1221809848.233708.1381110727421.JavaMail.root@mailhub059.itcs.purdue.edu>

Exactly
----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "Matthew Knepley" <knepley at gmail.com>
Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 9:49:59 PM
Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration


   Because he is using other stuff in mkl besides blas/lapack!

On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
> 
> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
> 
> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.

   Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.

   Barry

> 
> Is there a reason you do not want to use --with-blas-lapack-lib?
> 
>   Thanks,
> 
>     Matt
>  
> Thanks
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

From balay at mcs.anl.gov  Sun Oct  6 20:57:50 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Sun, 6 Oct 2013 20:57:50 -0500 (CDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <1221809848.233708.1381110727421.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1221809848.233708.1381110727421.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <alpine.LFD.2.03.1310062054240.18289@mcs.anl.gov>

If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
etc - then things will break.

So somehow these two options should work together - and perhaps the
way they can do is within blaslapack.py [and not a separate mkl.py]

Satish

On Sun, 6 Oct 2013, Jose David Bermeol wrote:

> Exactly
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Matthew Knepley" <knepley at gmail.com>
> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:49:59 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> 
>    Because he is using other stuff in mkl besides blas/lapack!
> 
> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> > On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> > Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
> > 
> > -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
> > 
> > Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
> 
>    Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
> 
>    Barry
> 
> > 
> > Is there a reason you do not want to use --with-blas-lapack-lib?
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > Thanks
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 


From jbermeol at purdue.edu  Sun Oct  6 21:07:31 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 22:07:31 -0400 (EDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <alpine.LFD.2.03.1310062054240.18289@mcs.anl.gov>
Message-ID: <1678798108.233743.1381111651106.JavaMail.root@mailhub059.itcs.purdue.edu>

Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  

----- Original Message -----
From: "Satish Balay" <balay at mcs.anl.gov>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 9:57:50 PM
Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration

If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
etc - then things will break.

So somehow these two options should work together - and perhaps the
way they can do is within blaslapack.py [and not a separate mkl.py]

Satish

On Sun, 6 Oct 2013, Jose David Bermeol wrote:

> Exactly
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Matthew Knepley" <knepley at gmail.com>
> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:49:59 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> 
>    Because he is using other stuff in mkl besides blas/lapack!
> 
> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> > On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> > Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
> > 
> > -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
> > 
> > Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
> 
>    Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
> 
>    Barry
> 
> > 
> > Is there a reason you do not want to use --with-blas-lapack-lib?
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > Thanks
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 

From jedbrown at mcs.anl.gov  Sun Oct  6 21:12:20 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 06 Oct 2013 21:12:20 -0500
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <alpine.LFD.2.03.1310062054240.18289@mcs.anl.gov>
References: <1221809848.233708.1381110727421.JavaMail.root@mailhub059.itcs.purdue.edu>
	<alpine.LFD.2.03.1310062054240.18289@mcs.anl.gov>
Message-ID: <874n8tvngb.fsf@mcs.anl.gov>

Satish Balay <balay at mcs.anl.gov> writes:

> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
> etc - then things will break.
>
> So somehow these two options should work together - and perhaps the
> way they can do is within blaslapack.py [and not a separate mkl.py]

Note that recent MKL has mkl_link_tool that we should use to avoid
hard-coding the looping strategies needed to link MKL.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131006/d5c3f53d/attachment.pgp>

From balay at mcs.anl.gov  Sun Oct  6 21:30:30 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Sun, 6 Oct 2013 21:30:30 -0500 (CDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <1678798108.233743.1381111651106.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1678798108.233743.1381111651106.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <alpine.LFD.2.03.1310062116250.18289@mcs.anl.gov>

On Sun, 6 Oct 2013, Jose David Bermeol wrote:

> Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. 

Perhaps there is a misunderstanding here.

> > > -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5


$ nm -Ao libmkl_intel_lp64.a |egrep "( pardiso_getenv_| dgeev_| ddot_)" |grep ' T '
libmkl_intel_lp64.a:_pardiso_getenv_lp64.o:0000000000000000 T pardiso_getenv_
libmkl_intel_lp64.a:_pardiso_getenv_lp64.o:0000000000000000 T pardiso_getenv__
libmkl_intel_lp64.a:_dgeev_lp64.o:0000000000000000 T dgeev_
libmkl_intel_lp64.a:_ddot_lp64.o:0000000000000000 T ddot_

Here -lmkl_intel_lp64 has pardiso, blas, lapack.

So when petsc blaslapack.py from petsc configure picks up something
else [other than mkl] for blaslapakck - that will be a conflict.

i.e if one links with '-lmkl_intel_lp64 -llapak -lblas' its a conflict.

So somehow petsc configure should avoid getting into this situation..

Satish

> So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  
> 
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:57:50 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
> etc - then things will break.
> 
> So somehow these two options should work together - and perhaps the
> way they can do is within blaslapack.py [and not a separate mkl.py]
> 
> Satish
> 
> On Sun, 6 Oct 2013, Jose David Bermeol wrote:
> 
> > Exactly
> > ----- Original Message -----
> > From: "Barry Smith" <bsmith at mcs.anl.gov>
> > To: "Matthew Knepley" <knepley at gmail.com>
> > Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> > Sent: Sunday, October 6, 2013 9:49:59 PM
> > Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> > 
> > 
> >    Because he is using other stuff in mkl besides blas/lapack!
> > 
> > On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > 
> > > On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> > > Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
> > > 
> > > -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
> > > 
> > > Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
> > 
> >    Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
> > 
> >    Barry
> > 
> > > 
> > > Is there a reason you do not want to use --with-blas-lapack-lib?
> > > 
> > >   Thanks,
> > > 
> > >     Matt
> > >  
> > > Thanks
> > > 
> > > 
> > > 
> > > -- 
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > 
> 


From bsmith at mcs.anl.gov  Sun Oct  6 21:54:53 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 6 Oct 2013 21:54:53 -0500
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <1678798108.233743.1381111651106.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1678798108.233743.1381111651106.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <030D574D-893F-458D-B098-6B2594B731F9@mcs.anl.gov>


   Jose,

    Since you are currently working on interfacing cpardiso I would just have a cpardiso.py (or cpardiso_mkl.py) (and NO mkl.py) that manages/lists the libraries needed for cpardiso. Don't have both a cpardiso.py and cpardiso_mkl.py. 

    Since, as Satish reports, cpardiso.py brings in also the mkl blas/lapack I would also have the cpardiso.py script check that the blas/lapack libraries being linked are compatible with the cpardiso mkl libraries being brought in and provide a useful error message telling them how to set the blas/lapack libraries to fix the problem when needed.

   We definitely do not want to muddy up the already horrible blaslapack.py code by introducing the concept of cpardiso there.

    Anyways, once you've simplified things, just make a pull request at bitbucket and we can try it and help with the kinks.


   Barry



On Oct 6, 2013, at 9:07 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  
> 
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:57:50 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
> etc - then things will break.
> 
> So somehow these two options should work together - and perhaps the
> way they can do is within blaslapack.py [and not a separate mkl.py]
> 
> Satish
> 
> On Sun, 6 Oct 2013, Jose David Bermeol wrote:
> 
>> Exactly
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "Matthew Knepley" <knepley at gmail.com>
>> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Sunday, October 6, 2013 9:49:59 PM
>> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
>> 
>> 
>>   Because he is using other stuff in mkl besides blas/lapack!
>> 
>> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>>> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
>>> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
>>> 
>>> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
>>> 
>>> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
>> 
>>   Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
>> 
>>   Barry
>> 
>>> 
>>> Is there a reason you do not want to use --with-blas-lapack-lib?
>>> 
>>>  Thanks,
>>> 
>>>    Matt
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 


From jbermeol at purdue.edu  Sun Oct  6 22:03:49 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 23:03:49 -0400 (EDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <030D574D-893F-458D-B098-6B2594B731F9@mcs.anl.gov>
Message-ID: <635244856.233835.1381115028883.JavaMail.root@mailhub059.itcs.purdue.edu>

Yes, blaslapack.py is enough big and horrible to add more code. Well the easier configuration is not a high priority think for me(but it something that in some point I will have to do). So for now I will have a basic cpardiso.py and then I will try to ensure that the blas/lapack libraries are correct.

----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 10:54:53 PM
Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration


   Jose,

    Since you are currently working on interfacing cpardiso I would just have a cpardiso.py (or cpardiso_mkl.py) (and NO mkl.py) that manages/lists the libraries needed for cpardiso. Don't have both a cpardiso.py and cpardiso_mkl.py. 

    Since, as Satish reports, cpardiso.py brings in also the mkl blas/lapack I would also have the cpardiso.py script check that the blas/lapack libraries being linked are compatible with the cpardiso mkl libraries being brought in and provide a useful error message telling them how to set the blas/lapack libraries to fix the problem when needed.

   We definitely do not want to muddy up the already horrible blaslapack.py code by introducing the concept of cpardiso there.

    Anyways, once you've simplified things, just make a pull request at bitbucket and we can try it and help with the kinks.


   Barry



On Oct 6, 2013, at 9:07 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  
> 
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:57:50 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
> etc - then things will break.
> 
> So somehow these two options should work together - and perhaps the
> way they can do is within blaslapack.py [and not a separate mkl.py]
> 
> Satish
> 
> On Sun, 6 Oct 2013, Jose David Bermeol wrote:
> 
>> Exactly
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "Matthew Knepley" <knepley at gmail.com>
>> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Sunday, October 6, 2013 9:49:59 PM
>> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
>> 
>> 
>>   Because he is using other stuff in mkl besides blas/lapack!
>> 
>> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>>> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
>>> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
>>> 
>>> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
>>> 
>>> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
>> 
>>   Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
>> 
>>   Barry
>> 
>>> 
>>> Is there a reason you do not want to use --with-blas-lapack-lib?
>>> 
>>>  Thanks,
>>> 
>>>    Matt
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 

From jbermeol at purdue.edu  Sun Oct  6 22:22:18 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 6 Oct 2013 23:22:18 -0400 (EDT)
Subject: [petsc-users] [petsc-dev]  Superlu_dist error
In-Reply-To: <C53AEC14-5A29-425E-9A2C-0586BEEE61C5@mcs.anl.gov>
Message-ID: <1140845902.233878.1381116138062.JavaMail.root@mailhub059.itcs.purdue.edu>

No it doesn't

Nonzeros in L       10
	Nonzeros in U       10
	nonzeros in L+U     10
	nonzeros in LSUB    10
        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
                              5.91278e-05 / 5.91278e-05 / 5.91278e-05
	EQUIL time             0.00
	ROWPERM time           0.00
	COLPERM time           0.00
	SYMBFACT time          0.00
	DISTRIBUTE time        0.00
	FACTOR time            0.00
	Factor flops	1.000000e+02	Mflops 	    0.29
	SOLVE time             0.00
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx_ABglobal line 228 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./test_superlu_dist on a arch-linux2-cxx-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 23:19:49 2013
[0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Sun Oct  6 22:05:55 2013
[0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl COPTFLAGS=-O CXXOPTFLAGS=-O FOPTFLAGS=-O --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 4:44:32 PM
Subject: Re: [petsc-dev] [petsc-users] Superlu_dist error


  If you install with -O instead of -O3 (use a different PETSC_ARCH so it doesn't overwrite) does the problem go away?

   Barry

On Oct 6, 2013, at 12:52 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Hi again, now I'm getting the error again. The problem happens when I'm using the flag -O3 for the compiler. So what should I do next to solve this??
> 
> Attached is my code, and for tis example I'm creating the matrix in the code.
> 
> Thanks  
> 
> Nonzeros in L       10
> 	Nonzeros in U       10
> 	nonzeros in L+U     10
> 	nonzeros in LSUB    10
> 	NUMfact space (MB) sum(procs):  L\U	0.00	all	0.03
> 	Total highmark (MB):  All	0.03	Avg	0.02	Max	0.02
>        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
>                              0.000124216 / 4.81606e-05 / 8.61883e-05
> 	EQUIL time             0.00
> 	ROWPERM time           0.00
> 	COLPERM time           0.00
> 	SYMBFACT time          0.00
> 	DISTRIBUTE time        0.00
> 	FACTOR time            0.00
> 	Factor flops	1.000000e+02	Mflops 	    0.31
> 	SOLVE time             0.00
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [1]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Signal received!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [1]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [1]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
> [0]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-03 CXXOPTFLAGS=-03 FOPTFLAGS=-03
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>, petsc-users at mcs.anl.gov
> Sent: Sunday, October 6, 2013 8:19:30 AM
> Subject: Re: [petsc-users] Superlu_dist error
> 
> 
> On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
> 
> 
> 
> 
> Hi again, I compile with the following configuration: 
> --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis 
> 
> 
> 
> Get rid of: 
> 
> 
> --with-pic=1 --with-fortran=1 --with-fortran-kernels=0 
> 
> 
> since they do not really do anything, and just put back MKL. I suspect you will get a crash, and then 
> it sounds like an MKL bug, or a bizarre incompatibility between SuperLU and MKL. If not, we explore further. 
> 
> 
> Thanks, 
> 
> 
> Matt 
> 
> 
> And my code run perfec, so that means is a MKL problem or a mismatch between versions, so how to test what is the problem?? 
> 
> Thanks 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "Jose David Bermeol" < jbermeol at purdue.edu > 
> Cc: "For users of the development version of PETSc" < petsc-dev at mcs.anl.gov >, petsc-users at mcs.anl.gov 
> Sent: Saturday, October 5, 2013 11:55:24 PM 
> Subject: Re: [petsc-users] Superlu_dist error 
> 
> 
> On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
> 
> 
> 
> 
> Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening. 
> 
> 
> 
> So it looks like SuperLU_Dist is bombing during an LAPACK operation. It could be an MKL problem, or a SuperLU_Dist problem, or our problem, 
> or a mismatch between versions. I would try to simplify the configuration in order to cut down on the possibilities. Eliminate everything that is not 
> necessary for SuperLU_dist first. Then change to --download-f-blas-lapack. If you still have a crash, send us the matrix since that should be 
> reproducible and we can report a SuperLU_dist bug or fix our code. 
> 
> 
> Thanks, 
> 
> 
> Matt 
> 
> 
> Thanks 
> 
> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed 
> Nonzeros in L 10 
> Nonzeros in U 10 
> nonzeros in L+U 10 
> nonzeros in LSUB 10 
> NUMfact space (MB) sum(procs): L\U 0.00 all 0.03 
> Total highmark (MB): All 0.03 Avg 0.02 Max 0.02 
> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
> 0.000146866 / 0.000145912 / 0.000146389 
> EQUIL time 0.00 
> ROWPERM time 0.00 
> COLPERM time 0.00 
> SYMBFACT time 0.00 
> DISTRIBUTE time 0.00 
> FACTOR time 0.00 
> Factor flops 1.000000e+02 Mflops 0.31 
> SOLVE time 0.00 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger 
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
> Try option -start_in_debugger or -on_error_attach_debugger 
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
> [0]PETSC ERROR: likely location of problem given in stack below 
> [1]PETSC ERROR: likely location of problem given in stack below 
> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
> [0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of the function 
> [1]PETSC ERROR: is given. 
> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [1]PETSC ERROR: INSTEAD the line number of the start of the function 
> [0]PETSC ERROR: is given. 
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ 
> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
> Signal received! 
> [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [1]PETSC ERROR: See docs/changes/index.html for recent updates. 
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. 
> [1]PETSC ERROR: See docs/index.html for manual pages. 
> [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
> [1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
> Signal received! 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates. 
> [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
> See docs/faq.html for hints about trouble shooting. 
> [0]PETSC ERROR: See docs/index.html for manual pages. 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
> [0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 
> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
> [0]PETSC ERROR: ------------------------------------------------------------------------ 
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener<test_superlu_dist.c>

From sonyablade2010 at hotmail.com  Mon Oct  7 00:54:09 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Mon, 7 Oct 2013 06:54:09 +0100
Subject: [petsc-users] Installation Error
In-Reply-To: <CAMYG4G=uqsp4ZL1gdDRfuEmWV2t7kjsYDr_51AnBJcUfhH90Xg@mail.gmail.com>
References: <DUB119-W3036AF2259016F4489C700C8110@phx.gbl>,
	<alpine.LFD.2.03.1310042346140.2061@mcs.anl.gov>,
	<DUB119-W477957D10254798F6F31FAC8110@phx.gbl>,
	<CAMYG4GnyuGVYcRh_jnXOz1g+QeH5C4gq9nnWdo_mQj2uR+ZprQ@mail.gmail.com>,
	<alpine.LFD.2.03.1310050928090.2061@mcs.anl.gov>,
	<DUB119-W2AB34817078825E15C781C8120@phx.gbl>,
	<CAMYG4G=uqsp4ZL1gdDRfuEmWV2t7kjsYDr_51AnBJcUfhH90Xg@mail.gmail.com>
Message-ID: <DUB119-W15FCD75A7828CF95EE3F13C8130@phx.gbl>

> 1) You must use the same compiler that you configured with. 
> 
> 2) You can get all the libraries you need to specify using 
> 
> make get linklibs 
> 
> Matt 
> 

All errors have been eliminated now, but this time compiler throws below exception at the run time?

"During startup program exited with code 0xc000007b."


Google search reveals that this is x32/x64 conflict and I'm sure that I've used?
Cygwin_32 version for Petsc and Slepc libararies. I attach the dependency walker?
snapshot view when the program is run. ?


Any idea on that? 		 	   		  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dependency walker.jpg
Type: image/jpeg
Size: 333822 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/e914d298/attachment-0001.jpg>

From christophe.ortiz at ciemat.es  Mon Oct  7 03:03:09 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Mon, 7 Oct 2013 10:03:09 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
Message-ID: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>

Hi all

I need some suggestions to design a code with PETSc.

I want to solve a 1D problem composed of several diffusion equations and a
lot of ODEs (thousands).

- After discretization, the diffusion equations have terms in i-1, i and
i+1.

- The diffusion equations have additional terms due to couping with all the
ODEs. These terms are non-linear.

- The terms of the ODEs are local (only depend on node i) but are
non-linear.

Any advice to design the code ?

How should I distribute the terms between IFunction and RHSFunction ?

Any special attention to DMDA ? Should I declare just one DMDA with dof >>1
?

Many thanks in advance !
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/46c583f5/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct  7 06:56:24 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Oct 2013 06:56:24 -0500
Subject: [petsc-users] [petsc-dev]  Superlu_dist error
In-Reply-To: <1140845902.233878.1381116138062.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1140845902.233878.1381116138062.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <0E75BEAB-B050-4465-BF20-0FADD3996B4F@mcs.anl.gov>


  Run with -start_in_debugger noxterm (since it seems to crash on one process) and type where when it crashes in the debugger. Also could run in valgrind http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind


On Oct 6, 2013, at 10:22 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> No it doesn't
> 
> Nonzeros in L       10
> 	Nonzeros in U       10
> 	nonzeros in L+U     10
> 	nonzeros in LSUB    10
>        Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
>                              5.91278e-05 / 5.91278e-05 / 5.91278e-05
> 	EQUIL time             0.00
> 	ROWPERM time           0.00
> 	COLPERM time           0.00
> 	SYMBFACT time          0.00
> 	DISTRIBUTE time        0.00
> 	FACTOR time            0.00
> 	Factor flops	1.000000e+02	Mflops 	    0.29
> 	SOLVE time             0.00
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx_ABglobal line 228 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./test_superlu_dist on a arch-linux2-cxx-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 23:19:49 2013
> [0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/compile_pardiso/petsc-3.4.2/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Sun Oct  6 22:05:55 2013
> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl COPTFLAGS=-O CXXOPTFLAGS=-O FOPTFLAGS=-O --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 4:44:32 PM
> Subject: Re: [petsc-dev] [petsc-users] Superlu_dist error
> 
> 
>  If you install with -O instead of -O3 (use a different PETSC_ARCH so it doesn't overwrite) does the problem go away?
> 
>   Barry
> 
> On Oct 6, 2013, at 12:52 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> 
>> Hi again, now I'm getting the error again. The problem happens when I'm using the flag -O3 for the compiler. So what should I do next to solve this??
>> 
>> Attached is my code, and for tis example I'm creating the matrix in the code.
>> 
>> Thanks  
>> 
>> Nonzeros in L       10
>> 	Nonzeros in U       10
>> 	nonzeros in L+U     10
>> 	nonzeros in LSUB    10
>> 	NUMfact space (MB) sum(procs):  L\U	0.00	all	0.03
>> 	Total highmark (MB):  All	0.03	Avg	0.02	Max	0.02
>>       Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
>>                             0.000124216 / 4.81606e-05 / 8.61883e-05
>> 	EQUIL time             0.00
>> 	ROWPERM time           0.00
>> 	COLPERM time           0.00
>> 	SYMBFACT time          0.00
>> 	DISTRIBUTE time        0.00
>> 	FACTOR time            0.00
>> 	Factor flops	1.000000e+02	Mflops 	    0.31
>> 	SOLVE time             0.00
>> [1]PETSC ERROR: ------------------------------------------------------------------------
>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [1]PETSC ERROR: likely location of problem given in stack below
>> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [1]PETSC ERROR:       is given.
>> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>> [1]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [1]PETSC ERROR: Signal received!
>> [1]PETSC ERROR: ------------------------------------------------------------------------
>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
>> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [1]PETSC ERROR: See docs/index.html for manual pages.
>> [1]PETSC ERROR: ------------------------------------------------------------------------
>> [1]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
>> [1]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
>> [1]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
>> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
>> [1]PETSC ERROR: ------------------------------------------------------------------------
>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
>> Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>> [0]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
>> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/petsc/petsc_superlu_dist/src/mat/interface/matrix.c
>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./test_solver on a arch-linux2-c-debug named carter-fe00.rcac.purdue.edu by jbermeol Sun Oct  6 13:43:00 2013
>> [0]PETSC ERROR: Libraries linked from /home/jbermeol/petsc/petsc_superlu_dist/arch-linux2-c-debug/lib
>> [0]PETSC ERROR: Configure run at Sun Oct  6 13:38:20 2013
>> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis COPTFLAGS=-03 CXXOPTFLAGS=-03 FOPTFLAGS=-03
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>> 
>> ----- Original Message -----
>> From: "Matthew Knepley" <knepley at gmail.com>
>> To: "Jose David Bermeol" <jbermeol at purdue.edu>
>> Cc: "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>, petsc-users at mcs.anl.gov
>> Sent: Sunday, October 6, 2013 8:19:30 AM
>> Subject: Re: [petsc-users] Superlu_dist error
>> 
>> 
>> On Sun, Oct 6, 2013 at 12:10 AM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
>> 
>> 
>> 
>> 
>> Hi again, I compile with the following configuration: 
>> --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --download-f-blas-lapack --download-superlu_dist=yes --download-superlu=yes --download-parmetis=yes --download-metis 
>> 
>> 
>> 
>> Get rid of: 
>> 
>> 
>> --with-pic=1 --with-fortran=1 --with-fortran-kernels=0 
>> 
>> 
>> since they do not really do anything, and just put back MKL. I suspect you will get a crash, and then 
>> it sounds like an MKL bug, or a bizarre incompatibility between SuperLU and MKL. If not, we explore further. 
>> 
>> 
>> Thanks, 
>> 
>> 
>> Matt 
>> 
>> 
>> And my code run perfec, so that means is a MKL problem or a mismatch between versions, so how to test what is the problem?? 
>> 
>> Thanks 
>> 
>> ----- Original Message ----- 
>> From: "Matthew Knepley" < knepley at gmail.com > 
>> To: "Jose David Bermeol" < jbermeol at purdue.edu > 
>> Cc: "For users of the development version of PETSc" < petsc-dev at mcs.anl.gov >, petsc-users at mcs.anl.gov 
>> Sent: Saturday, October 5, 2013 11:55:24 PM 
>> Subject: Re: [petsc-users] Superlu_dist error 
>> 
>> 
>> On Sat, Oct 5, 2013 at 10:49 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 
>> 
>> 
>> 
>> 
>> Hi I'm runnig petsc trying to solve a linear system with superlu_dist. However i have a memory violation, atached is the code, and here is the output. Email me if you need something else to figured out what is happening. 
>> 
>> 
>> 
>> So it looks like SuperLU_Dist is bombing during an LAPACK operation. It could be an MKL problem, or a SuperLU_Dist problem, or our problem, 
>> or a mismatch between versions. I would try to simplify the configuration in order to cut down on the possibilities. Eliminate everything that is not 
>> necessary for SuperLU_dist first. Then change to --download-f-blas-lapack. If you still have a crash, send us the matrix since that should be 
>> reproducible and we can report a SuperLU_dist bug or fix our code. 
>> 
>> 
>> Thanks, 
>> 
>> 
>> Matt 
>> 
>> 
>> Thanks 
>> 
>> mpiexec -n 2 ./test_solver -mat_superlu_dist_statprint -mat_superlu_dist_matinput distributed 
>> Nonzeros in L 10 
>> Nonzeros in U 10 
>> nonzeros in L+U 10 
>> nonzeros in LSUB 10 
>> NUMfact space (MB) sum(procs): L\U 0.00 all 0.03 
>> Total highmark (MB): All 0.03 Avg 0.02 Max 0.02 
>> Mat conversion(PETSc->SuperLU_DIST) time (max/min/avg): 
>> 0.000146866 / 0.000145912 / 0.000146389 
>> EQUIL time 0.00 
>> ROWPERM time 0.00 
>> COLPERM time 0.00 
>> SYMBFACT time 0.00 
>> DISTRIBUTE time 0.00 
>> FACTOR time 0.00 
>> Factor flops 1.000000e+02 Mflops 0.31 
>> SOLVE time 0.00 
>> [0]PETSC ERROR: ------------------------------------------------------------------------ 
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger 
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range 
>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
>> Try option -start_in_debugger or -on_error_attach_debugger 
>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors 
>> [0]PETSC ERROR: likely location of problem given in stack below 
>> [1]PETSC ERROR: likely location of problem given in stack below 
>> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
>> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ 
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, 
>> [0]PETSC ERROR: [1]PETSC ERROR: INSTEAD the line number of the start of the function 
>> [1]PETSC ERROR: is given. 
>> [1]PETSC ERROR: [1] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
>> [1]PETSC ERROR: [1] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
>> [1]PETSC ERROR: INSTEAD the line number of the start of the function 
>> [0]PETSC ERROR: is given. 
>> [0]PETSC ERROR: [0] SuperLU_DIST:pzgssvx line 234 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
>> [0]PETSC ERROR: [1] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
>> [1]PETSC ERROR: --------------------- Error Message ------------------------------------ 
>> [1]PETSC ERROR: [0] MatMatSolve_SuperLU_DIST line 198 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c 
>> [0]PETSC ERROR: [0] MatMatSolve line 3207 /home/jbermeol/Nemo5/libs/petsc/build-cplx/src/mat/interface/matrix.c 
>> Signal received! 
>> [1]PETSC ERROR: ------------------------------------------------------------------------ 
>> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
>> [1]PETSC ERROR: See docs/changes/index.html for recent updates. 
>> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. 
>> [1]PETSC ERROR: See docs/index.html for manual pages. 
>> [1]PETSC ERROR: ------------------------------------------------------------------------ 
>> [1]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
>> [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------ 
>> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
>> [1]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
>> [1]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
>> Signal received! 
>> [0]PETSC ERROR: ------------------------------------------------------------------------ 
>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. 
>> [0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------------------------------ 
>> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
>> See docs/faq.html for hints about trouble shooting. 
>> [0]PETSC ERROR: See docs/index.html for manual pages. 
>> [0]PETSC ERROR: ------------------------------------------------------------------------ 
>> [0]PETSC ERROR: ./test_solver on a linux-complex named carter-fe02.rcac.purdue.edu by jbermeol Sat Oct 5 23:45:21 2013 
>> [0]PETSC ERROR: Libraries linked from /home/jbermeol/Nemo5/libs/petsc/build-cplx/linux-complex/lib 
>> [0]PETSC ERROR: Configure run at Sat Oct 5 11:19:36 2013 
>> [0]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1 
>> Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-shared-libraries=1 --with-debugging=1 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-valgrind-dir=/apps/rhel6/valgrind/3.8.1 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-mkl-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-mkl-lib="[/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_lp64.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_intel_thread.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_core.so,/apps/rhel6/intel/composer_xe_2013.3.163/mkl/../compiler/lib/intel64/libiomp5.so]" --with-cpardiso-dir=/home/jbermeol/testPetscSolvers/intel_mkl_cpardiso --with-hdf5 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-superlu=yes --download-mumps=yes --download-spooles=yes --download-pastix=yes --download-ptscotch=yes --download-umfpack=yes --download-sowing 
>> [0]PETSC ERROR: ------------------------------------------------------------------------ 
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file 
>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener<test_superlu_dist.c>


From bsmith at mcs.anl.gov  Mon Oct  7 07:00:59 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Oct 2013 07:00:59 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
Message-ID: <162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>


  Chris,

    Take a look in the petsc branch barry/wirth-fusion-materials https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials  
in the file src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free to ask any questions.

    Barry

On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> Hi all
> 
> I need some suggestions to design a code with PETSc.
> 
> I want to solve a 1D problem composed of several diffusion equations and a lot of ODEs (thousands).
> 
> - After discretization, the diffusion equations have terms in i-1, i and i+1.
> 
> - The diffusion equations have additional terms due to couping with all the ODEs. These terms are non-linear.
> 
> - The terms of the ODEs are local (only depend on node i) but are non-linear.
> 
> Any advice to design the code ?
> 
> How should I distribute the terms between IFunction and RHSFunction ?
> 
> Any special attention to DMDA ? Should I declare just one DMDA with dof >>1 ?
> 
> Many thanks in advance !
> Christophe
> 


From asmund.ervik at ntnu.no  Mon Oct  7 07:30:30 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Mon, 07 Oct 2013 14:30:30 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <87d2nkzs5f.fsf@mcs.anl.gov>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov>
Message-ID: <5252A966.4010406@ntnu.no>

Hi again,

I've started to delve into the world of DMDA. At this point I want to
write code that goes as follows, but something tells me this may be a
bad idea? Synopsis of the code given below the snippet.

  ! The "array" array is allocated like this (taken from examples):
  ! call DMDAGetGhostCorners(SolScal,ib1,ibn,jb1,jbn,kb1,kbn,ierr)
  ! allocate(array(0:dof-1,ib1:ibn,jb1:jbn,kb1:kbn))

  subroutine petsc_to_local(da,vec,array,f,dof)
    DM, intent(in)                         :: da
    Vec,intent(in)                         :: vec
    PetscScalar, pointer, intent(inout)    :: array(:,:,:,:)
    integer,intent(in)                     :: dof
    real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof)
    !
    if (dof == 1) then
      call VecGetArrayF90(da,vec,array(0,:,:,:),ierr)
    else
      call VecGetArrayF90(da,vec,array,ierr)
    endif
    call transform_petsc_us(array,f,dof)
  end subroutine petsc_to_local
  subroutine transform_petsc_us(array,f,dof)
    !Note: assumed shape-array does the "coordinate transformation"
    real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
    real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
    integer :: n
    !
    do n=1,dof
      f(:,:,:,n) = array(n-1,:,:,:)
    enddo
  end subroutine transform_petsc_us

This is basically a wrapper around VecGetArrayF90 that does some things:
it handles dof=1 or higher in the same way, it reindexes dof to start at
1, moves the dof from the first to the last index (these two are for
compatibility with existing code), and perhaps most importantly it does
a "coordinate transformation" such that the end result (in the array f
that my code subsequently uses) is indexed from 1-stw (stw: stencil
width) to ilmax+stw (ilmax: local index i max).

This enables me to plug existing code straight in and manipulate the
array f. Then I do the opposite remap and call VecRestoreArrayF90, and
then DMLocalToGlobalBegin/End. The code between VecGet... and
VecRestore... will run for several seconds (or longer), it includes a
Poisson solve with KSP (I haven't figured that part out yet...)

Am I being stupid here, or is this a good solution? I'm guessing this
ends up making a copy into f, so it may hurt performance somewhat, but I
don't think that will be critical?

Best regards,
?smund


On 04. okt. 2013 22:40, Jed Brown wrote:
> Matthew Knepley <knepley at gmail.com> writes:
> 
>> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
>>
>>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
>>> hdf5 in the background.
>>>
>>
>> I guess not. It was just easier for me to get Xdmf going. We have trial
>> CGNS as well.
> 
> I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf gives
> you a little more flexibility in choosing a schema for the HDF5 file,
> but you can put other stuff in CGNS files and Xdmf has its share of
> shortcomings.  If you are happy with CGNS, I would say to use it and
> report any issues you have.  We'll enhance the implementation so it can
> do everything you need.
> 

From knepley at gmail.com  Mon Oct  7 07:39:35 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 7 Oct 2013 07:39:35 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <5252A966.4010406@ntnu.no>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
Message-ID: <CAMYG4GnPC7v=VN0AD10mb35Y6Af_UELvLvn90ejY4tO0hT7H8Q@mail.gmail.com>

On Mon, Oct 7, 2013 at 7:30 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

> Hi again,
>
> I've started to delve into the world of DMDA. At this point I want to
> write code that goes as follows, but something tells me this may be a
> bad idea? Synopsis of the code given below the snippet.
>
>   ! The "array" array is allocated like this (taken from examples):
>   ! call DMDAGetGhostCorners(SolScal,ib1,ibn,jb1,jbn,kb1,kbn,ierr)
>   ! allocate(array(0:dof-1,ib1:ibn,jb1:jbn,kb1:kbn))
>
>   subroutine petsc_to_local(da,vec,array,f,dof)
>     DM, intent(in)                         :: da
>     Vec,intent(in)                         :: vec
>     PetscScalar, pointer, intent(inout)    :: array(:,:,:,:)
>     integer,intent(in)                     :: dof
>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof)
>     !
>     if (dof == 1) then
>       call VecGetArrayF90(da,vec,array(0,:,:,:),ierr)
>     else
>       call VecGetArrayF90(da,vec,array,ierr)
>     endif
>     call transform_petsc_us(array,f,dof)
>   end subroutine petsc_to_local
>   subroutine transform_petsc_us(array,f,dof)
>     !Note: assumed shape-array does the "coordinate transformation"
>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
>     integer :: n
>     !
>     do n=1,dof
>       f(:,:,:,n) = array(n-1,:,:,:)
>     enddo
>   end subroutine transform_petsc_us
>
> This is basically a wrapper around VecGetArrayF90 that does some things:
> it handles dof=1 or higher in the same way, it reindexes dof to start at
> 1, moves the dof from the first to the last index (these two are for
> compatibility with existing code), and perhaps most importantly it does
> a "coordinate transformation" such that the end result (in the array f
> that my code subsequently uses) is indexed from 1-stw (stw: stencil
> width) to ilmax+stw (ilmax: local index i max).
>

You might want VecGetArrayDOFF90(). Its too bad F90 pointers cannot be
easily
manipulated like C or having a 1-offset would not require a copy.

    Matt


> This enables me to plug existing code straight in and manipulate the
> array f. Then I do the opposite remap and call VecRestoreArrayF90, and
> then DMLocalToGlobalBegin/End. The code between VecGet... and
> VecRestore... will run for several seconds (or longer), it includes a
> Poisson solve with KSP (I haven't figured that part out yet...)
>
> Am I being stupid here, or is this a good solution? I'm guessing this
> ends up making a copy into f, so it may hurt performance somewhat, but I
> don't think that will be critical?
>
> Best regards,
> ?smund
>
>
> On 04. okt. 2013 22:40, Jed Brown wrote:
> > Matthew Knepley <knepley at gmail.com> writes:
> >
> >> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no>
> wrote:
> >>
> >>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
> >>> hdf5 in the background.
> >>>
> >>
> >> I guess not. It was just easier for me to get Xdmf going. We have trial
> >> CGNS as well.
> >
> > I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf gives
> > you a little more flexibility in choosing a schema for the HDF5 file,
> > but you can put other stuff in CGNS files and Xdmf has its share of
> > shortcomings.  If you are happy with CGNS, I would say to use it and
> > report any issues you have.  We'll enhance the implementation so it can
> > do everything you need.
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/7dd2ad1d/attachment.html>

From asmund.ervik at ntnu.no  Mon Oct  7 07:59:17 2013
From: asmund.ervik at ntnu.no (=?ISO-8859-1?Q?=C5smund_Ervik?=)
Date: Mon, 07 Oct 2013 14:59:17 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <CAMYG4GnPC7v=VN0AD10mb35Y6Af_UELvLvn90ejY4tO0hT7H8Q@mail.gmail.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>	<87d2nkzs5f.fsf@mcs.anl.gov>	<5252A966.4010406@ntnu.no>
	<CAMYG4GnPC7v=VN0AD10mb35Y6Af_UELvLvn90ejY4tO0hT7H8Q@mail.gmail.com>
Message-ID: <5252B025.4050500@ntnu.no>

Matt,

I don't think there exists such a subroutine:

$ cd petsc-dev/src
$ ack "GetArrayDOF" | wc -l
51
$ ack "GetArrayDOFF90" | wc -l
0

?smund


On 07. okt. 2013 14:39, Matthew Knepley wrote:
> On Mon, Oct 7, 2013 at 7:30 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:
> 
>> Hi again,
>>
>> I've started to delve into the world of DMDA. At this point I want to
>> write code that goes as follows, but something tells me this may be a
>> bad idea? Synopsis of the code given below the snippet.
>>
>>   ! The "array" array is allocated like this (taken from examples):
>>   ! call DMDAGetGhostCorners(SolScal,ib1,ibn,jb1,jbn,kb1,kbn,ierr)
>>   ! allocate(array(0:dof-1,ib1:ibn,jb1:jbn,kb1:kbn))
>>
>>   subroutine petsc_to_local(da,vec,array,f,dof)
>>     DM, intent(in)                         :: da
>>     Vec,intent(in)                         :: vec
>>     PetscScalar, pointer, intent(inout)    :: array(:,:,:,:)
>>     integer,intent(in)                     :: dof
>>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof)
>>     !
>>     if (dof == 1) then
>>       call VecGetArrayF90(da,vec,array(0,:,:,:),ierr)
>>     else
>>       call VecGetArrayF90(da,vec,array,ierr)
>>     endif
>>     call transform_petsc_us(array,f,dof)
>>   end subroutine petsc_to_local
>>   subroutine transform_petsc_us(array,f,dof)
>>     !Note: assumed shape-array does the "coordinate transformation"
>>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
>>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
>>     integer :: n
>>     !
>>     do n=1,dof
>>       f(:,:,:,n) = array(n-1,:,:,:)
>>     enddo
>>   end subroutine transform_petsc_us
>>
>> This is basically a wrapper around VecGetArrayF90 that does some things:
>> it handles dof=1 or higher in the same way, it reindexes dof to start at
>> 1, moves the dof from the first to the last index (these two are for
>> compatibility with existing code), and perhaps most importantly it does
>> a "coordinate transformation" such that the end result (in the array f
>> that my code subsequently uses) is indexed from 1-stw (stw: stencil
>> width) to ilmax+stw (ilmax: local index i max).
>>
> 
> You might want VecGetArrayDOFF90(). Its too bad F90 pointers cannot be
> easily
> manipulated like C or having a 1-offset would not require a copy.
> 
>     Matt
> 
> 
>> This enables me to plug existing code straight in and manipulate the
>> array f. Then I do the opposite remap and call VecRestoreArrayF90, and
>> then DMLocalToGlobalBegin/End. The code between VecGet... and
>> VecRestore... will run for several seconds (or longer), it includes a
>> Poisson solve with KSP (I haven't figured that part out yet...)
>>
>> Am I being stupid here, or is this a good solution? I'm guessing this
>> ends up making a copy into f, so it may hurt performance somewhat, but I
>> don't think that will be critical?
>>
>> Best regards,
>> ?smund
>>
>>
>> On 04. okt. 2013 22:40, Jed Brown wrote:
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>
>>>> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no>
>> wrote:
>>>>
>>>>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
>>>>> hdf5 in the background.
>>>>>
>>>>
>>>> I guess not. It was just easier for me to get Xdmf going. We have trial
>>>> CGNS as well.
>>>
>>> I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf gives
>>> you a little more flexibility in choosing a schema for the HDF5 file,
>>> but you can put other stuff in CGNS files and Xdmf has its share of
>>> shortcomings.  If you are happy with CGNS, I would say to use it and
>>> report any issues you have.  We'll enhance the implementation so it can
>>> do everything you need.
>>>
>>
> 
> 
> 

From knepley at gmail.com  Mon Oct  7 08:02:54 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 7 Oct 2013 08:02:54 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <5252B025.4050500@ntnu.no>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<CAMYG4GnPC7v=VN0AD10mb35Y6Af_UELvLvn90ejY4tO0hT7H8Q@mail.gmail.com>
	<5252B025.4050500@ntnu.no>
Message-ID: <CAMYG4G=JXvjP8jFg3yo1o4X_ieqgoB4rHcZ=ED8VeXhjibbdSQ@mail.gmail.com>

On Mon, Oct 7, 2013 at 7:59 AM, ?smund Ervik <asmund.ervik at ntnu.no>wrotMatt,

>
> I don't think there exists such a subroutine:
>
> $ cd petsc-dev/src
> $ ack "GetArrayDOF" | wc -l
> 51
> $ ack "GetArrayDOFF90" | wc -l
> 0
>

That must have happened in the F90 rewrite. The DMDAVecGetArrayDOF() makes
another
dimension for the dof. It would be easy to write the F90 wrapper if someone
is going to use it.

   Matt


> ?smund
>
>
> On 07. okt. 2013 14:39, Matthew Knepley wrote:
> > On Mon, Oct 7, 2013 at 7:30 AM, ?smund Ervik <asmund.ervik at ntnu.no>
> wrote:
> >
> >> Hi again,
> >>
> >> I've started to delve into the world of DMDA. At this point I want to
> >> write code that goes as follows, but something tells me this may be a
> >> bad idea? Synopsis of the code given below the snippet.
> >>
> >>   ! The "array" array is allocated like this (taken from examples):
> >>   ! call DMDAGetGhostCorners(SolScal,ib1,ibn,jb1,jbn,kb1,kbn,ierr)
> >>   ! allocate(array(0:dof-1,ib1:ibn,jb1:jbn,kb1:kbn))
> >>
> >>   subroutine petsc_to_local(da,vec,array,f,dof)
> >>     DM, intent(in)                         :: da
> >>     Vec,intent(in)                         :: vec
> >>     PetscScalar, pointer, intent(inout)    :: array(:,:,:,:)
> >>     integer,intent(in)                     :: dof
> >>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof)
> >>     !
> >>     if (dof == 1) then
> >>       call VecGetArrayF90(da,vec,array(0,:,:,:),ierr)
> >>     else
> >>       call VecGetArrayF90(da,vec,array,ierr)
> >>     endif
> >>     call transform_petsc_us(array,f,dof)
> >>   end subroutine petsc_to_local
> >>   subroutine transform_petsc_us(array,f,dof)
> >>     !Note: assumed shape-array does the "coordinate transformation"
> >>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
> >>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
> >>     integer :: n
> >>     !
> >>     do n=1,dof
> >>       f(:,:,:,n) = array(n-1,:,:,:)
> >>     enddo
> >>   end subroutine transform_petsc_us
> >>
> >> This is basically a wrapper around VecGetArrayF90 that does some things:
> >> it handles dof=1 or higher in the same way, it reindexes dof to start at
> >> 1, moves the dof from the first to the last index (these two are for
> >> compatibility with existing code), and perhaps most importantly it does
> >> a "coordinate transformation" such that the end result (in the array f
> >> that my code subsequently uses) is indexed from 1-stw (stw: stencil
> >> width) to ilmax+stw (ilmax: local index i max).
> >>
> >
> > You might want VecGetArrayDOFF90(). Its too bad F90 pointers cannot be
> > easily
> > manipulated like C or having a 1-offset would not require a copy.
> >
> >     Matt
> >
> >
> >> This enables me to plug existing code straight in and manipulate the
> >> array f. Then I do the opposite remap and call VecRestoreArrayF90, and
> >> then DMLocalToGlobalBegin/End. The code between VecGet... and
> >> VecRestore... will run for several seconds (or longer), it includes a
> >> Poisson solve with KSP (I haven't figured that part out yet...)
> >>
> >> Am I being stupid here, or is this a good solution? I'm guessing this
> >> ends up making a copy into f, so it may hurt performance somewhat, but I
> >> don't think that will be critical?
> >>
> >> Best regards,
> >> ?smund
> >>
> >>
> >> On 04. okt. 2013 22:40, Jed Brown wrote:
> >>> Matthew Knepley <knepley at gmail.com> writes:
> >>>
> >>>> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no>
> >> wrote:
> >>>>
> >>>>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
> >>>>> hdf5 in the background.
> >>>>>
> >>>>
> >>>> I guess not. It was just easier for me to get Xdmf going. We have
> trial
> >>>> CGNS as well.
> >>>
> >>> I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf
> gives
> >>> you a little more flexibility in choosing a schema for the HDF5 file,
> >>> but you can put other stuff in CGNS files and Xdmf has its share of
> >>> shortcomings.  If you are happy with CGNS, I would say to use it and
> >>> report any issues you have.  We'll enhance the implementation so it can
> >>> do everything you need.
> >>>
> >>
> >
> >
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/d2c19db5/attachment.html>

From asmund.ervik at ntnu.no  Mon Oct  7 08:06:53 2013
From: asmund.ervik at ntnu.no (=?ISO-8859-1?Q?=C5smund_Ervik?=)
Date: Mon, 07 Oct 2013 15:06:53 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <CAMYG4G=JXvjP8jFg3yo1o4X_ieqgoB4rHcZ=ED8VeXhjibbdSQ@mail.gmail.com>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>	<87d2nkzs5f.fsf@mcs.anl.gov>	<5252A966.4010406@ntnu.no>	<CAMYG4GnPC7v=VN0AD10mb35Y6Af_UELvLvn90ejY4tO0hT7H8Q@mail.gmail.com>	<5252B025.4050500@ntnu.no>
	<CAMYG4G=JXvjP8jFg3yo1o4X_ieqgoB4rHcZ=ED8VeXhjibbdSQ@mail.gmail.com>
Message-ID: <5252B1ED.7010601@ntnu.no>

On 07. okt. 2013 15:02, Matthew Knepley wrote:
> 
> That must have happened in the F90 rewrite. The DMDAVecGetArrayDOF() makes
> another
> dimension for the dof. It would be easy to write the F90 wrapper if someone
> is going to use it.

My question was not really on this point. DMDAVecGetArrayF90() works
with both dof=1 and dof>1, you just have to pass in different arrays,
that's no biggie.

The question was really about my re-mapping of coordinates from global
to local. Is what I wrote a good way to do it, or are there better ways?

?smund

> 
>    Matt
> 
> 
>> ?smund
>>
>>
>> On 07. okt. 2013 14:39, Matthew Knepley wrote:
>>> On Mon, Oct 7, 2013 at 7:30 AM, ?smund Ervik <asmund.ervik at ntnu.no>
>> wrote:
>>>
>>>> Hi again,
>>>>
>>>> I've started to delve into the world of DMDA. At this point I want to
>>>> write code that goes as follows, but something tells me this may be a
>>>> bad idea? Synopsis of the code given below the snippet.
>>>>
>>>>   ! The "array" array is allocated like this (taken from examples):
>>>>   ! call DMDAGetGhostCorners(SolScal,ib1,ibn,jb1,jbn,kb1,kbn,ierr)
>>>>   ! allocate(array(0:dof-1,ib1:ibn,jb1:jbn,kb1:kbn))
>>>>
>>>>   subroutine petsc_to_local(da,vec,array,f,dof)
>>>>     DM, intent(in)                         :: da
>>>>     Vec,intent(in)                         :: vec
>>>>     PetscScalar, pointer, intent(inout)    :: array(:,:,:,:)
>>>>     integer,intent(in)                     :: dof
>>>>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof)
>>>>     !
>>>>     if (dof == 1) then
>>>>       call VecGetArrayF90(da,vec,array(0,:,:,:),ierr)
>>>>     else
>>>>       call VecGetArrayF90(da,vec,array,ierr)
>>>>     endif
>>>>     call transform_petsc_us(array,f,dof)
>>>>   end subroutine petsc_to_local
>>>>   subroutine transform_petsc_us(array,f,dof)
>>>>     !Note: assumed shape-array does the "coordinate transformation"
>>>>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
>>>>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
>>>>     integer :: n
>>>>     !
>>>>     do n=1,dof
>>>>       f(:,:,:,n) = array(n-1,:,:,:)
>>>>     enddo
>>>>   end subroutine transform_petsc_us
>>>>
>>>> This is basically a wrapper around VecGetArrayF90 that does some things:
>>>> it handles dof=1 or higher in the same way, it reindexes dof to start at
>>>> 1, moves the dof from the first to the last index (these two are for
>>>> compatibility with existing code), and perhaps most importantly it does
>>>> a "coordinate transformation" such that the end result (in the array f
>>>> that my code subsequently uses) is indexed from 1-stw (stw: stencil
>>>> width) to ilmax+stw (ilmax: local index i max).
>>>>
>>>
>>> You might want VecGetArrayDOFF90(). Its too bad F90 pointers cannot be
>>> easily
>>> manipulated like C or having a 1-offset would not require a copy.
>>>
>>>     Matt
>>>
>>>
>>>> This enables me to plug existing code straight in and manipulate the
>>>> array f. Then I do the opposite remap and call VecRestoreArrayF90, and
>>>> then DMLocalToGlobalBegin/End. The code between VecGet... and
>>>> VecRestore... will run for several seconds (or longer), it includes a
>>>> Poisson solve with KSP (I haven't figured that part out yet...)
>>>>
>>>> Am I being stupid here, or is this a good solution? I'm guessing this
>>>> ends up making a copy into f, so it may hurt performance somewhat, but I
>>>> don't think that will be critical?
>>>>
>>>> Best regards,
>>>> ?smund
>>>>
>>>>
>>>> On 04. okt. 2013 22:40, Jed Brown wrote:
>>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>>
>>>>>> On Fri, Oct 4, 2013 at 2:10 PM, ?smund Ervik <asmund.ervik at ntnu.no>
>>>> wrote:
>>>>>>
>>>>>>>  Is there any reason I should prefer Xdmf over Cgns? I think both use
>>>>>>> hdf5 in the background.
>>>>>>>
>>>>>>
>>>>>> I guess not. It was just easier for me to get Xdmf going. We have
>> trial
>>>>>> CGNS as well.
>>>>>
>>>>> I don't think we have a CGNS writer, but DMPlex can read it.  Xdmf
>> gives
>>>>> you a little more flexibility in choosing a schema for the HDF5 file,
>>>>> but you can put other stuff in CGNS files and Xdmf has its share of
>>>>> shortcomings.  If you are happy with CGNS, I would say to use it and
>>>>> report any issues you have.  We'll enhance the implementation so it can
>>>>> do everything you need.
>>>>>
>>>>
>>>
>>>
>>>
>>
> 
> 
> 

From christophe.ortiz at ciemat.es  Mon Oct  7 08:16:41 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Mon, 7 Oct 2013 15:16:41 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
Message-ID: <CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>

Hi Barry,

Thanks for the reply.

I had a look at the example ex10.c. But I see it does not fully corresponds
to what one can see in the branch. For instance, in the example that is
found in the tutorials, there is no RHS defined, no Jacobian, just the
IFunction.

Could you please send me the latest version of ex10.c ? Many thanks in
advance.

Christophe

CIEMAT
Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
Unidad de Materiales
Edificio 2 - Planta 0 - Despacho 28m
Avenida Complutense 40,
28040 Madrid, Spain
Tel: +34 91496 2582
Fax: +34 91346 6442

--
Q
Por favor, piense en el medio ambiente antes de imprimir este mensaje.
Please consider the environment before printing this email.


On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Chris,
>
>     Take a look in the petsc branch barry/wirth-fusion-materials
> https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
> in the file src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c
>   feel free to ask any questions.
>
>     Barry
>
> On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> > Hi all
> >
> > I need some suggestions to design a code with PETSc.
> >
> > I want to solve a 1D problem composed of several diffusion equations and
> a lot of ODEs (thousands).
> >
> > - After discretization, the diffusion equations have terms in i-1, i and
> i+1.
> >
> > - The diffusion equations have additional terms due to couping with all
> the ODEs. These terms are non-linear.
> >
> > - The terms of the ODEs are local (only depend on node i) but are
> non-linear.
> >
> > Any advice to design the code ?
> >
> > How should I distribute the terms between IFunction and RHSFunction ?
> >
> > Any special attention to DMDA ? Should I declare just one DMDA with dof
> >>1 ?
> >
> > Many thanks in advance !
> > Christophe
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/348fc509/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct  7 08:23:52 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Oct 2013 08:23:52 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
Message-ID: <E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>


  I have put it into the next branch. So just download petsc from the bitbucket site with git and then do git checkout next 

   The code won't work with the current PETSc release.

On Oct 7, 2013, at 8:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> Hi Barry,
> 
> Thanks for the reply.
> 
> I had a look at the example ex10.c. But I see it does not fully corresponds to what one can see in the branch. For instance, in the example that is found in the tutorials, there is no RHS defined, no Jacobian, just the IFunction.
> 
> Could you please send me the latest version of ex10.c ? Many thanks in advance.
> 
> Christophe
> 
> CIEMAT
> Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> Unidad de Materiales
> Edificio 2 - Planta 0 - Despacho 28m
> Avenida Complutense 40,
> 28040 Madrid, Spain
> Tel: +34 91496 2582
> Fax: +34 91346 6442
> 
> --
> Q 
> Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> Please consider the environment before printing this email.
> 
> 
> On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Chris,
> 
>     Take a look in the petsc branch barry/wirth-fusion-materials https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
> in the file src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free to ask any questions.
> 
>     Barry
> 
> On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> 
> > Hi all
> >
> > I need some suggestions to design a code with PETSc.
> >
> > I want to solve a 1D problem composed of several diffusion equations and a lot of ODEs (thousands).
> >
> > - After discretization, the diffusion equations have terms in i-1, i and i+1.
> >
> > - The diffusion equations have additional terms due to couping with all the ODEs. These terms are non-linear.
> >
> > - The terms of the ODEs are local (only depend on node i) but are non-linear.
> >
> > Any advice to design the code ?
> >
> > How should I distribute the terms between IFunction and RHSFunction ?
> >
> > Any special attention to DMDA ? Should I declare just one DMDA with dof >>1 ?
> >
> > Many thanks in advance !
> > Christophe
> >
> 
> 


From jedbrown at mcs.anl.gov  Mon Oct  7 10:14:37 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 10:14:37 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <5252A966.4010406@ntnu.no>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
Message-ID: <87d2nht8o2.fsf@mcs.anl.gov>

?smund Ervik <asmund.ervik at ntnu.no> writes:
>   subroutine transform_petsc_us(array,f,dof)
>     !Note: assumed shape-array does the "coordinate transformation"
>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f

This is a really bad ordering for cache-based architectures because as
you traverse the mesh, you need to pull values from segments of the
array that are far apart.  We have yet to encounter a practical project
in which indexing the first way (dofs adjacent in memory) is not faster,
sometimes by large factors.  (For example, Clawpack was recently
convinced to change and saw a 50% speedup.  The PETSc-FUN3D papers had
many comparisons.)  I recommend transforming your code to use the former
ordering, or if you must, use CPP macros so that you can change it at
compile time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/58fcfa86/attachment.pgp>

From asmund.ervik at ntnu.no  Mon Oct  7 10:40:23 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Mon, 07 Oct 2013 17:40:23 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <87d2nht8o2.fsf@mcs.anl.gov>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<87d2nht8o2.fsf@mcs.anl.gov>
Message-ID: <5252D5E7.8050906@ntnu.no>

Jed,

On ma. 07. okt. 2013 kl. 17.14 +0200, Jed Brown wrote:
> ?smund Ervik <asmund.ervik at ntnu.no> writes:
>>   subroutine transform_petsc_us(array,f,dof)
>>     !Note: assumed shape-array does the "coordinate transformation"
>>     real, intent(in), dimension(dof,1-stw:,1-stw:,1-stw:) :: array
>>     real,intent(inout),dimension(1-stw:,1-stw:1-stw:,dof) :: f
>
> This is a really bad ordering for cache-based architectures because as
> you traverse the mesh, you need to pull values from segments of the
> array that are far apart.  We have yet to encounter a practical project
> in which indexing the first way (dofs adjacent in memory) is not faster,
> sometimes by large factors.  (For example, Clawpack was recently
> convinced to change and saw a 50% speedup.  The PETSc-FUN3D papers had
> many comparisons.)  I recommend transforming your code to use the former
> ordering, or if you must, use CPP macros so that you can change it at
> compile time.

Thanks for this comment; I have been suspecting this is a bad idea, 
nice to get confirmation. I will follow your advice.

Other than that, does the mapping from local to global using Fortran 
assumed-shape arrays make sense, or is there a better way of doing it?

?smund



From jedbrown at mcs.anl.gov  Mon Oct  7 10:55:27 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 10:55:27 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <5252D5E7.8050906@ntnu.no>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<87d2nht8o2.fsf@mcs.anl.gov> <5252D5E7.8050906@ntnu.no>
Message-ID: <877gdpt6s0.fsf@mcs.anl.gov>

?smund Ervik <asmund.ervik at ntnu.no> writes:
> Thanks for this comment; I have been suspecting this is a bad idea, 
> nice to get confirmation. I will follow your advice.

Cool.

> Other than that, does the mapping from local to global using Fortran 
> assumed-shape arrays make sense, or is there a better way of doing it?

It looks okay to me.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/41d836a0/attachment.pgp>

From jedbrown at mcs.anl.gov  Mon Oct  7 11:14:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 11:14:09 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
Message-ID: <8738odt5wu.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> I want to solve a 1D problem composed of several diffusion equations and a
> lot of ODEs (thousands).
>
> - After discretization, the diffusion equations have terms in i-1, i and
> i+1.
>
> - The diffusion equations have additional terms due to couping with all the
> ODEs. These terms are non-linear.
>
> - The terms of the ODEs are local (only depend on node i) but are
> non-linear.

How do you want to extend your code?  Is it acceptable to have a
centralized struct where you place all variables, or should separate
modules be able to "register" additional variables?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/6fc43a9b/attachment.pgp>

From yye00 at tacc.utexas.edu  Mon Oct  7 12:57:41 2013
From: yye00 at tacc.utexas.edu (Yaakoub El Khamra)
Date: Mon, 7 Oct 2013 12:57:41 -0500
Subject: [petsc-users] ex44f.F90 with fortran data types running with 2 cores
Message-ID: <CAGBp19BHwCW-90kmaTaX_KfR9LN1AqYUuZH2aXVHCYQN-dRpxg@mail.gmail.com>

I am going over ex44f.F90 with petsc-3.4.2 built with fortran data types
and compiled with intel compilers. Minor code modifications were made:
type(Vec) as opposed to Vec and so on. I also added CHKERRQ(ierr) wherever
it needed to go and increased the DMDA size (I used 32).

The ierr value at the KSPSolve was 78, and stdout had the error message
below. I must be doing something wrong. Anyone has any thoughts on what
that might be?

[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[1]PETSC ERROR: PetscMallocValidate: error detected at
 PetscSignalHandlerDefault() line 149 in
/work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/error/signal.c
[1]PETSC ERROR: Memory at address 0x89cfa0 is corrupted
[1]PETSC ERROR: Probably write past beginning or end of array
[1]PETSC ERROR: Last intact block allocated in PetscObjectListAdd() line
119 in /work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/objects/olist.c
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Memory corruption!
[1]PETSC ERROR:  !
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR:
/work/02453/saswata/yye00/src/petsc-3.4.2/src/ksp/ksp/examples/tutorials/ex44f
on a arch-linux2-cxx-debug named c545-501.stampede.tacc.utexas.edu by
saswata Mon Oct  7 12:52:32 2013
[1]PETSC ERROR: Libraries linked from /work/02453/saswata/yye00/lib
[1]PETSC ERROR: Configure run at Mon Oct  7 11:27:02 2013
[1]PETSC ERROR: Configure options --prefix=/work/02453/saswata/yye00/
--with-log=1 --with-info=1 --with-ctable=1 --with-clanguage=C++
--with-fortran-datatypes=1 --with-fortran=1 --with-debugging=1
--with-debugger=1 --with-shared-libraries=1 --with-debugger=gdb
--with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64
--with-mpi=1 --with-mpi-compilers=1 --with-x=0 --with-sieve=1
--download-scientificpython --download-fiat --download-generator
--download-triangle --with-ctetgen --download-chaco --download-boost=1
--with-parmetis=1 --download-parmetis=1 --download-fiat=1
--download-scalapack=1 --download-hypre=1 --download-pastix=1
--download-superlu_dist=1 --download-mumps=1 --download-exodusii=1
--download-netcdf=1 --download-hdf5=1 --download-moab=1
--download-umfpack=1 --download-metis --download-ctetgen --download-ptscotch
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: PetscMallocValidate() line 137 in
/work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/memory/mtr.c
[1]PETSC ERROR: PetscSignalHandlerDefault() line 149 in
/work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/error/signal.c
[cli_1]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 0) - process 1
^C[c545-501.stampede.tacc.utexas.edu:mpirun_rsh][signal_processor] Caught
signal 2, killing job
TACC: MPI job exited with code: 1

TACC: Shutdown complete. Exiting.




Regards
Yaakoub El Khamra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/f61e63c6/attachment.html>

From jedbrown at mcs.anl.gov  Mon Oct  7 13:49:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 13:49:09 -0500
Subject: [petsc-users] ex44f.F90 with fortran data types running with 2
	cores
In-Reply-To: <CAGBp19BHwCW-90kmaTaX_KfR9LN1AqYUuZH2aXVHCYQN-dRpxg@mail.gmail.com>
References: <CAGBp19BHwCW-90kmaTaX_KfR9LN1AqYUuZH2aXVHCYQN-dRpxg@mail.gmail.com>
Message-ID: <87pprgsyqi.fsf@mcs.anl.gov>

Yaakoub El Khamra <yye00 at tacc.utexas.edu> writes:

> I am going over ex44f.F90 with petsc-3.4.2 built with fortran data types
> and compiled with intel compilers. Minor code modifications were made:
> type(Vec) as opposed to Vec 

This example is meant to be usable with normal variable declarations
(i.e., without data types).  Unfortunately, the same code can't work
both ways.

> and so on. I also added CHKERRQ(ierr) wherever it needed to go and
> increased the DMDA size (I used 32).
>
> The ierr value at the KSPSolve was 78, 

That is not a valid return code so memory was corrupt by that time.  I
recommend following the instructions and using Valgrind to find the bug.

> and stdout had the error message below. I must be doing something
> wrong. Anyone has any thoughts on what that might be?
>
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR:
> or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [1]PETSC ERROR: PetscMallocValidate: error detected at
>  PetscSignalHandlerDefault() line 149 in
> /work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/error/signal.c
> [1]PETSC ERROR: Memory at address 0x89cfa0 is corrupted
> [1]PETSC ERROR: Probably write past beginning or end of array
> [1]PETSC ERROR: Last intact block allocated in PetscObjectListAdd() line
> 119 in /work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/objects/olist.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Memory corruption!
> [1]PETSC ERROR:  !
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR:
> /work/02453/saswata/yye00/src/petsc-3.4.2/src/ksp/ksp/examples/tutorials/ex44f
> on a arch-linux2-cxx-debug named c545-501.stampede.tacc.utexas.edu by
> saswata Mon Oct  7 12:52:32 2013
> [1]PETSC ERROR: Libraries linked from /work/02453/saswata/yye00/lib
> [1]PETSC ERROR: Configure run at Mon Oct  7 11:27:02 2013
> [1]PETSC ERROR: Configure options --prefix=/work/02453/saswata/yye00/
> --with-log=1 --with-info=1 --with-ctable=1 --with-clanguage=C++
> --with-fortran-datatypes=1 --with-fortran=1 --with-debugging=1
> --with-debugger=1 --with-shared-libraries=1 --with-debugger=gdb
> --with-blas-lapack-dir=/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64
> --with-mpi=1 --with-mpi-compilers=1 --with-x=0 --with-sieve=1
> --download-scientificpython --download-fiat --download-generator
> --download-triangle --with-ctetgen --download-chaco --download-boost=1
> --with-parmetis=1 --download-parmetis=1 --download-fiat=1
> --download-scalapack=1 --download-hypre=1 --download-pastix=1
> --download-superlu_dist=1 --download-mumps=1 --download-exodusii=1
> --download-netcdf=1 --download-hdf5=1 --download-moab=1
> --download-umfpack=1 --download-metis --download-ctetgen --download-ptscotch
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscMallocValidate() line 137 in
> /work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/memory/mtr.c
> [1]PETSC ERROR: PetscSignalHandlerDefault() line 149 in
> /work/02453/saswata/yye00/src/petsc-3.4.2/src/sys/error/signal.c
> [cli_1]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 0) - process 1
> ^C[c545-501.stampede.tacc.utexas.edu:mpirun_rsh][signal_processor] Caught
> signal 2, killing job
> TACC: MPI job exited with code: 1
>
> TACC: Shutdown complete. Exiting.
>
>
>
>
> Regards
> Yaakoub El Khamra
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/32924bc8/attachment.pgp>

From pengxwang at hotmail.com  Mon Oct  7 14:08:38 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Mon, 7 Oct 2013 14:08:38 -0500
Subject: [petsc-users] questions about KSP with-pc_type hypre
Message-ID: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>

Hello,

   I am trying to solve a linear system with KSP for a 3-D domain.  The computational domain is a cubic space with Neumann condition on the lateral sides( y, and z directions).  A smaller round plate is on the left side in x direction.  The right side is a rectangular plane either with or without a hole.  In testing studies, the KSP solver works fine both with and without -pc_type hypre runtime option for the domain with a whole plane(without a hole) on the right, but diverges for the case of a plane with a hole with -ps_type hypre option. 

  My question is whether the hypre pre-conditioner is sensitive to the geometry boundary?  If the answer is yes, is there any other pre-conditioner available for my particular case(the domain with a plane which with a hole on the right side) to reduce the number of iterations? Thanks in advance.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/e6200d87/attachment.html>

From irving at naml.us  Mon Oct  7 14:11:40 2013
From: irving at naml.us (Geoffrey Irving)
Date: Mon, 7 Oct 2013 12:11:40 -0700
Subject: [petsc-users] stiff support in src/ts/examples/tutorials/ex9.c
Message-ID: <C236A2DB-A27E-4BF0-9A04-C67545E8A789@naml.us>

Does src/ts/examples/tutorials/ex9.c support implicit methods for stiff problems?  It doesn't explicitly create any linear solvers, but maybe these are created automatically if you pick the appropriate TS?  That would presumably require derivative information though, which I don't see.

Thanks,
Geoffrey
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/e670751e/attachment.pgp>

From irving at naml.us  Mon Oct  7 14:14:58 2013
From: irving at naml.us (Geoffrey Irving)
Date: Mon, 7 Oct 2013 12:14:58 -0700
Subject: [petsc-users] stiff support in src/ts/examples/tutorials/ex9.c
In-Reply-To: <C236A2DB-A27E-4BF0-9A04-C67545E8A789@naml.us>
References: <C236A2DB-A27E-4BF0-9A04-C67545E8A789@naml.us>
Message-ID: <D96155D2-1885-4ED1-87A9-516A34695989@naml.us>

On Oct 7, 2013, at 12:11 PM, Geoffrey Irving <irving at naml.us> wrote:

> Does src/ts/examples/tutorials/ex9.c support implicit methods for stiff problems?  It doesn't explicitly create any linear solvers, but maybe these are created automatically if you pick the appropriate TS?  That would presumably require derivative information though, which I don't see.
> 
> Thanks,
> Geoffrey

Oops, I missed the FVIJacobian function, which would certainly indicate stiff support.

Geoffrey
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/814979fb/attachment-0001.pgp>

From christophe.ortiz at ciemat.es  Mon Oct  7 14:49:39 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Mon, 7 Oct 2013 21:49:39 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
Message-ID: <CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>

Thanks. I got it.

For the moment I have some general questions.

- I see that in this example all terms--diffusion and reactions--are put
into RHSFunction. There is no IFunction. Is there a reason for that ?  In
which cases one should use IFunction or RHSFunction ? In ts/.../ex25.c,
diffusion terms are defined in IFunction and reaction terms in RHSFunction.
Why not here ?

- If only RHSFunction is used, how is it possible to define boundary
conditions ? For instance, a Dirichlet boundary condition, u(i=0)=3. In
ts/.../ex2.c, only RHSFunction is used and the boundary conditions are
defined through u_t,

- In this type of problem, which TS scheme is recommended ? TSARKIMEX ?

- The option TSARKIMEXSetFullyImplicit is used. Why ? Does it help ?

Christophe

CIEMAT
Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
Unidad de Materiales
Edificio 2 - Planta 0 - Despacho 28m
Avenida Complutense 40,
28040 Madrid, Spain
Tel: +34 91496 2582
Fax: +34 91346 6442

--
Q
Por favor, piense en el medio ambiente antes de imprimir este mensaje.
Please consider the environment before printing this email.


On Mon, Oct 7, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I have put it into the next branch. So just download petsc from the
> bitbucket site with git and then do git checkout next
>
>    The code won't work with the current PETSc release.
>
> On Oct 7, 2013, at 8:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> > Hi Barry,
> >
> > Thanks for the reply.
> >
> > I had a look at the example ex10.c. But I see it does not fully
> corresponds to what one can see in the branch. For instance, in the example
> that is found in the tutorials, there is no RHS defined, no Jacobian, just
> the IFunction.
> >
> > Could you please send me the latest version of ex10.c ? Many thanks in
> advance.
> >
> > Christophe
> >
> > CIEMAT
> > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > Unidad de Materiales
> > Edificio 2 - Planta 0 - Despacho 28m
> > Avenida Complutense 40,
> > 28040 Madrid, Spain
> > Tel: +34 91496 2582
> > Fax: +34 91346 6442
> >
> > --
> > Q
> > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > Please consider the environment before printing this email.
> >
> >
> > On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Chris,
> >
> >     Take a look in the petsc branch barry/wirth-fusion-materials
> https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
> > in the file
> src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free
> to ask any questions.
> >
> >     Barry
> >
> > On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
> >
> > > Hi all
> > >
> > > I need some suggestions to design a code with PETSc.
> > >
> > > I want to solve a 1D problem composed of several diffusion equations
> and a lot of ODEs (thousands).
> > >
> > > - After discretization, the diffusion equations have terms in i-1, i
> and i+1.
> > >
> > > - The diffusion equations have additional terms due to couping with
> all the ODEs. These terms are non-linear.
> > >
> > > - The terms of the ODEs are local (only depend on node i) but are
> non-linear.
> > >
> > > Any advice to design the code ?
> > >
> > > How should I distribute the terms between IFunction and RHSFunction ?
> > >
> > > Any special attention to DMDA ? Should I declare just one DMDA with
> dof >>1 ?
> > >
> > > Many thanks in advance !
> > > Christophe
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/a1b90380/attachment.html>

From christophe.ortiz at ciemat.es  Mon Oct  7 14:51:35 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Mon, 7 Oct 2013 21:51:35 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <8738odt5wu.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<8738odt5wu.fsf@mcs.anl.gov>
Message-ID: <CANBrw+4XcKR94OxpcxEEB7oND19rb8gieOAzRs8sdHo0p5S-Kg@mail.gmail.com>

On Mon, Oct 7, 2013 at 6:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > I want to solve a 1D problem composed of several diffusion equations and
> a
> > lot of ODEs (thousands).
> >
> > - After discretization, the diffusion equations have terms in i-1, i and
> > i+1.
> >
> > - The diffusion equations have additional terms due to couping with all
> the
> > ODEs. These terms are non-linear.
> >
> > - The terms of the ODEs are local (only depend on node i) but are
> > non-linear.
>
>
Hi Jed,

I am not sure to understand what you mean. What do you mean by modules able
to register additional variables ?

Christophe


> How do you want to extend your code?  Is it acceptable to have a
> centralized struct where you place all variables, or should separate
> modules be able to "register" additional variables?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/83cb815c/attachment.html>

From mfadams at lbl.gov  Mon Oct  7 15:16:30 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Mon, 7 Oct 2013 16:16:30 -0400
Subject: [petsc-users] questions about KSP with-pc_type hypre
In-Reply-To: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>
References: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>
Message-ID: <53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>

I assume this is a Lapacian and its not a pure Neumann problem.  I did not quite follow you description.  Do you have some Dirichlet BCs?

Hypre is a very robust solver, for a scalable solver.  ILU, the default PC, is very robust but slow (that's why its the default).  You could try gamg, another AMG solver to get another data point.

On Oct 7, 2013, at 3:08 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Hello,
> 
>    I am trying to solve a linear system with KSP for a 3-D domain.  The computational domain is a cubic space with Neumann condition on the lateral sides( y, and z directions).  A smaller round plate is on the left side in x direction.  The right side is a rectangular plane either with or without a hole.  In testing studies, the KSP solver works fine both with and without -pc_type hypre runtime option for the domain with a whole plane(without a hole) on the right, but diverges for the case of a plane with a hole with -ps_type hypre option. 
> 
>   My question is whether the hypre pre-conditioner is sensitive to the geometry boundary?  If the answer is yes, is there any other pre-conditioner available for my particular case(the domain with a plane which with a hole on the right side) to reduce the number of iterations? Thanks in advance.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/c7c4bab5/attachment.html>

From pengxwang at hotmail.com  Mon Oct  7 15:28:47 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Mon, 7 Oct 2013 15:28:47 -0500
Subject: [petsc-users] questions about KSP with-pc_type hypre
In-Reply-To: <53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>
References: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>,
	<53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>
Message-ID: <BAY179-W843968F8DE0BE9372F27F4B4130@phx.gbl>

Sorry for the mis-info. 

Yes, I have Dirichlet BCs. The  smaller round plate on the left side and the rectangular plane on the right size are the Dirichlet BCs. And the inside the hole of the rectangular plane is Neumann BCs.  I am just wondering the Neumann BCs of the hole have some effect which lead to divergence.  

I will try gamg as your suggestion. Thanks.


Subject: Re: [petsc-users] questions about KSP with-pc_type hypre
From: mfadams at lbl.gov
Date: Mon, 7 Oct 2013 16:16:30 -0400
CC: petsc-users at mcs.anl.gov
To: pengxwang at hotmail.com

I assume this is a Lapacian and its not a pure Neumann problem.  I did not quite follow you description.  Do you have some Dirichlet BCs?
Hypre is a very robust solver, for a scalable solver.  ILU, the default PC, is very robust but slow (that's why its the default).  You could try gamg, another AMG solver to get another data point.
On Oct 7, 2013, at 3:08 PM, Roc Wang <pengxwang at hotmail.com> wrote:Hello,

   I am trying to solve a linear system with KSP for a 3-D domain.  The computational domain is a cubic space with Neumann condition on the lateral sides( y, and z directions).  A smaller round plate is on the left side in x direction.  The right side is a rectangular plane either with or without a hole.  In testing studies, the KSP solver works fine both with and without -pc_type hypre runtime option for the domain with a whole plane(without a hole) on the right, but diverges for the case of a plane with a hole with -ps_type hypre option. 

  My question is whether the hypre pre-conditioner is sensitive to the geometry boundary?  If the answer is yes, is there any other pre-conditioner available for my particular case(the domain with a plane which with a hole on the right side) to reduce the number of iterations? Thanks in advance.

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/0995d2ac/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct  7 16:10:10 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Oct 2013 16:10:10 -0500
Subject: [petsc-users] questions about KSP with-pc_type hypre
In-Reply-To: <BAY179-W843968F8DE0BE9372F27F4B4130@phx.gbl>
References: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>,
	<53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>
	<BAY179-W843968F8DE0BE9372F27F4B4130@phx.gbl>
Message-ID: <4DA5639B-F203-4B37-94C9-676716E89378@mcs.anl.gov>


On Oct 7, 2013, at 3:28 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Sorry for the mis-info. 
> 
> Yes, I have Dirichlet BCs. The smaller round plate on the left side and the rectangular plane on the right size are the Dirichlet BCs. And the inside the hole of the rectangular plane is Neumann BCs.  I am just wondering the Neumann BCs of the hole have some effect which lead to divergence.  

   "Some" Neumann boundary conditions shouldn't have much effect. Are you sure the Neumann BCs are applied correctly and that Neumann BCs are not being applied everywhere?  If Neumann BCs are applied almost everywhere on the boundary that can affect convergence.

   What happens with -pc_type lu?  Does it solve the linear system and give back answers you expect?


   Barry

> 
> I will try gamg as your suggestion. Thanks.
> 
> 
> Subject: Re: [petsc-users] questions about KSP with-pc_type hypre
> From: mfadams at lbl.gov
> Date: Mon, 7 Oct 2013 16:16:30 -0400
> CC: petsc-users at mcs.anl.gov
> To: pengxwang at hotmail.com
> 
> I assume this is a Lapacian and its not a pure Neumann problem.  I did not quite follow you description.  Do you have some Dirichlet BCs?
> 
> Hypre is a very robust solver, for a scalable solver.  ILU, the default PC, is very robust but slow (that's why its the default).  You could try gamg, another AMG solver to get another data point.
> 
> On Oct 7, 2013, at 3:08 PM, Roc Wang <pengxwang at hotmail.com> wrote:
> 
> Hello,
> 
>    I am trying to solve a linear system with KSP for a 3-D domain.  The computational domain is a cubic space with Neumann condition on the lateral sides( y, and z directions).  A smaller round plate is on the left side in x direction.  The right side is a rectangular plane either with or without a hole.  In testing studies, the KSP solver works fine both with and without -pc_type hypre runtime option for the domain with a whole plane(without a hole) on the right, but diverges for the case of a plane with a hole with -ps_type hypre option. 
> 
>   My question is whether the hypre pre-conditioner is sensitive to the geometry boundary?  If the answer is yes, is there any other pre-conditioner available for my particular case(the domain with a plane which with a hole on the right side) to reduce the number of iterations? Thanks in advance.
> 
> 


From bsmith at mcs.anl.gov  Mon Oct  7 16:14:21 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 7 Oct 2013 16:14:21 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
Message-ID: <747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>


On Oct 7, 2013, at 2:49 PM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> Thanks. I got it.
> 
> For the moment I have some general questions.
> 
> - I see that in this example all terms--diffusion and reactions--are put into RHSFunction. There is no IFunction. Is there a reason for that ?  In which cases one should use IFunction or RHSFunction ? In ts/.../ex25.c, diffusion terms are defined in IFunction and reaction terms in RHSFunction. Why not here ?

   RHS function is for non-stiff parts of the problem. IFunction for stiff parts. 

   There reason they are not separated in ex10.c is because I don't know for this problem what is what, so I just put them all in RHSFunction. But use a fully implicit solver.
> 
> - If only RHSFunction is used, how is it possible to define boundary conditions ? For instance, a Dirichlet boundary condition, u(i=0)=3. In ts/.../ex2.c, only RHSFunction is used and the boundary conditions are defined through u_t, 

   If, for example u is zero on the boundary for all time then you can just set u(t=0) to 0 on the boundary and u_t = 0 on the boundary and the boundary values of u will remain zero.  If you want to do something like u = f(t) on the boundary you can put that into the IFunction.
> 
> - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?

   Beats me.

> 
> - The option TSARKIMEXSetFullyImplicit is used. Why ? Does it help ?

    I just wanted to use a fully implicit solver; since I just used RHSFunction by default TSARKIMEX would treat it as explicit which I didn't want.

   Barry

> 
> Christophe
> 
> CIEMAT
> Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> Unidad de Materiales
> Edificio 2 - Planta 0 - Despacho 28m
> Avenida Complutense 40,
> 28040 Madrid, Spain
> Tel: +34 91496 2582
> Fax: +34 91346 6442
> 
> --
> Q 
> Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> Please consider the environment before printing this email.
> 
> 
> On Mon, Oct 7, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   I have put it into the next branch. So just download petsc from the bitbucket site with git and then do git checkout next
> 
>    The code won't work with the current PETSc release.
> 
> On Oct 7, 2013, at 8:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> 
> > Hi Barry,
> >
> > Thanks for the reply.
> >
> > I had a look at the example ex10.c. But I see it does not fully corresponds to what one can see in the branch. For instance, in the example that is found in the tutorials, there is no RHS defined, no Jacobian, just the IFunction.
> >
> > Could you please send me the latest version of ex10.c ? Many thanks in advance.
> >
> > Christophe
> >
> > CIEMAT
> > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > Unidad de Materiales
> > Edificio 2 - Planta 0 - Despacho 28m
> > Avenida Complutense 40,
> > 28040 Madrid, Spain
> > Tel: +34 91496 2582
> > Fax: +34 91346 6442
> >
> > --
> > Q
> > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > Please consider the environment before printing this email.
> >
> >
> > On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Chris,
> >
> >     Take a look in the petsc branch barry/wirth-fusion-materials https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
> > in the file src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free to ask any questions.
> >
> >     Barry
> >
> > On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> >
> > > Hi all
> > >
> > > I need some suggestions to design a code with PETSc.
> > >
> > > I want to solve a 1D problem composed of several diffusion equations and a lot of ODEs (thousands).
> > >
> > > - After discretization, the diffusion equations have terms in i-1, i and i+1.
> > >
> > > - The diffusion equations have additional terms due to couping with all the ODEs. These terms are non-linear.
> > >
> > > - The terms of the ODEs are local (only depend on node i) but are non-linear.
> > >
> > > Any advice to design the code ?
> > >
> > > How should I distribute the terms between IFunction and RHSFunction ?
> > >
> > > Any special attention to DMDA ? Should I declare just one DMDA with dof >>1 ?
> > >
> > > Many thanks in advance !
> > > Christophe
> > >
> >
> >
> 
> 


From jedbrown at mcs.anl.gov  Mon Oct  7 16:47:58 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 16:47:58 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4XcKR94OxpcxEEB7oND19rb8gieOAzRs8sdHo0p5S-Kg@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<8738odt5wu.fsf@mcs.anl.gov>
	<CANBrw+4XcKR94OxpcxEEB7oND19rb8gieOAzRs8sdHo0p5S-Kg@mail.gmail.com>
Message-ID: <8738ocsqgh.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> I am not sure to understand what you mean. What do you mean by modules able
> to register additional variables ?

It depends on the scope of your code.  If you are solving only one
problem, then you're probably happy to define a struct somewhere that
your entire code can see it and place your fields in there.  In that
case, you can use the offsetof() macro to do stable indexing within the
struct (so that you can add or reorder fields later without needing to
change code like matrix assembly).

Other projects are more like frameworks where users want to write
libraries of material models, process parametrizations, and the like.
If you had a single global struct, then everyone would be changing
things and you give up the ability to change the fields at run-time,
meaning that you'll have a soup of #ifdefs and in general, the struct
becomes a constant source of problems.  So you wouldn't want that.  A
better approach is to have a dynamic registration system where you
declare dependencies and access fields through an interface.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/d4271f03/attachment.pgp>

From jedbrown at mcs.anl.gov  Mon Oct  7 17:18:30 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 17:18:30 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
Message-ID: <87ob70rah5.fsf@mcs.anl.gov>

Barry Smith <bsmith at mcs.anl.gov> writes:
>> - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?
>
>    Beats me.

ARKIMEX should give you a decent integrator with adaptive error control.
Use '-ts_arkimex_type 1bee' to use backward Euler with an
extrapolation-based error estimator.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/e1237be1/attachment.pgp>

From jedbrown at mcs.anl.gov  Mon Oct  7 20:07:14 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 07 Oct 2013 20:07:14 -0500
Subject: [petsc-users] stiff support in src/ts/examples/tutorials/ex9.c
In-Reply-To: <D96155D2-1885-4ED1-87A9-516A34695989@naml.us>
References: <C236A2DB-A27E-4BF0-9A04-C67545E8A789@naml.us>
	<D96155D2-1885-4ED1-87A9-516A34695989@naml.us>
Message-ID: <874n8sr2nx.fsf@mcs.anl.gov>

Geoffrey Irving <irving at naml.us> writes:

> On Oct 7, 2013, at 12:11 PM, Geoffrey Irving <irving at naml.us> wrote:
>
>> Does src/ts/examples/tutorials/ex9.c support implicit methods for
>> stiff problems?  It doesn't explicitly create any linear solvers, but
>> maybe these are created automatically if you pick the appropriate TS?
>> That would presumably require derivative information though, which I
>> don't see.

The example assembles a block-diagonal matrix based on a cell-centered
characteristic decomposition for preconditioning.  I don't recall if it
was ever tested.  Better to just use coloring (really easy and not very
wasteful 1D problems).  Here is an example of a shallow water dam-break
using an implicit time integrator.

./ex9 -da_grid_x 100 -initial 0 -xmin -2 -xmax 2 -limit mc -physics shallow -bc_type outflow -ts_final_time 1 -ts_monitor_draw_solution -ts_type arkimex -ts_arkimex_fully_implicit -ts_adapt_type none -snes_converged_reason -ts_monitor -snes_fd_color -ts_max_snes_failures -1


Note that the problems in this example are non-stiff so implicit methods
won't be more efficient.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131007/6b2582e6/attachment.pgp>

From christophe.ortiz at ciemat.es  Tue Oct  8 03:57:09 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 8 Oct 2013 10:57:09 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
Message-ID: <CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>

On Mon, Oct 7, 2013 at 11:14 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Oct 7, 2013, at 2:49 PM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> > Thanks. I got it.
> >
> > For the moment I have some general questions.
> >
> > - I see that in this example all terms--diffusion and reactions--are put
> into RHSFunction. There is no IFunction. Is there a reason for that ?  In
> which cases one should use IFunction or RHSFunction ? In ts/.../ex25.c,
> diffusion terms are defined in IFunction and reaction terms in RHSFunction.
> Why not here ?
>
>    RHS function is for non-stiff parts of the problem. IFunction for stiff
> parts.
>
>    There reason they are not separated in ex10.c is because I don't know
> for this problem what is what, so I just put them all in RHSFunction. But
> use a fully implicit solver.
>

I understand.


> >
> > - If only RHSFunction is used, how is it possible to define boundary
> conditions ? For instance, a Dirichlet boundary condition, u(i=0)=3. In
> ts/.../ex2.c, only RHSFunction is used and the boundary conditions are
> defined through u_t,
>
>    If, for example u is zero on the boundary for all time then you can
> just set u(t=0) to 0 on the boundary and u_t = 0 on the boundary and the
> boundary values of u will remain zero.  If you want to do something like u
> = f(t) on the boundary you can put that into the IFunction.
>

I see. So I could put either everything in the IFunction, or everything in
RHS but the two boundaries that cannot be defined in RHS (u=f(t)), in
IFunction. Correct ?


> >
> > - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?
>
>    Beats me.
>
> >
> > - The option TSARKIMEXSetFullyImplicit is used. Why ? Does it help ?
>
>     I just wanted to use a fully implicit solver; since I just used
> RHSFunction by default TSARKIMEX would treat it as explicit which I didn't
> want.
>

It makes sense.

Thanks a lot. All this is clear now. I'll probably come back to you soon
with more specific questions about ordering and matrices.

Christophe


>
>    Barry
>
> >
> > Christophe
> >
> > CIEMAT
> > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > Unidad de Materiales
> > Edificio 2 - Planta 0 - Despacho 28m
> > Avenida Complutense 40,
> > 28040 Madrid, Spain
> > Tel: +34 91496 2582
> > Fax: +34 91346 6442
> >
> > --
> > Q
> > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > Please consider the environment before printing this email.
> >
> >
> > On Mon, Oct 7, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   I have put it into the next branch. So just download petsc from the
> bitbucket site with git and then do git checkout next
> >
> >    The code won't work with the current PETSc release.
> >
> > On Oct 7, 2013, at 8:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
> >
> > > Hi Barry,
> > >
> > > Thanks for the reply.
> > >
> > > I had a look at the example ex10.c. But I see it does not fully
> corresponds to what one can see in the branch. For instance, in the example
> that is found in the tutorials, there is no RHS defined, no Jacobian, just
> the IFunction.
> > >
> > > Could you please send me the latest version of ex10.c ? Many thanks in
> advance.
> > >
> > > Christophe
> > >
> > > CIEMAT
> > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> > > Unidad de Materiales
> > > Edificio 2 - Planta 0 - Despacho 28m
> > > Avenida Complutense 40,
> > > 28040 Madrid, Spain
> > > Tel: +34 91496 2582
> > > Fax: +34 91346 6442
> > >
> > > --
> > > Q
> > > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> > > Please consider the environment before printing this email.
> > >
> > >
> > > On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >   Chris,
> > >
> > >     Take a look in the petsc branch barry/wirth-fusion-materials
> https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
> > > in the file
> src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free
> to ask any questions.
> > >
> > >     Barry
> > >
> > > On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> > >
> > > > Hi all
> > > >
> > > > I need some suggestions to design a code with PETSc.
> > > >
> > > > I want to solve a 1D problem composed of several diffusion equations
> and a lot of ODEs (thousands).
> > > >
> > > > - After discretization, the diffusion equations have terms in i-1, i
> and i+1.
> > > >
> > > > - The diffusion equations have additional terms due to couping with
> all the ODEs. These terms are non-linear.
> > > >
> > > > - The terms of the ODEs are local (only depend on node i) but are
> non-linear.
> > > >
> > > > Any advice to design the code ?
> > > >
> > > > How should I distribute the terms between IFunction and RHSFunction ?
> > > >
> > > > Any special attention to DMDA ? Should I declare just one DMDA with
> dof >>1 ?
> > > >
> > > > Many thanks in advance !
> > > > Christophe
> > > >
> > >
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/1b665959/attachment-0001.html>

From christophe.ortiz at ciemat.es  Tue Oct  8 04:00:59 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 8 Oct 2013 11:00:59 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87ob70rah5.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<87ob70rah5.fsf@mcs.anl.gov>
Message-ID: <CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>

On Tue, Oct 8, 2013 at 12:18 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
> >> - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?
> >
> >    Beats me.
>
> ARKIMEX should give you a decent integrator with adaptive error control.
> Use '-ts_arkimex_type 1bee' to use backward Euler with an
> extrapolation-based error estimator.
>

Good to know. I tried TSBEULER but it has constant timestep.

-Is there any other TS with adaptive timestep ?

-With ARKIMEX, is there a way to control the timestep ? For instance, is it
possible to control the max factor between two successive timesteps (dt' =
factor*dt), in order to avoid rejections ?

- Is it possible to have Cranck-Nicholson with adaptive timestep ? I tried
TSCN but it seems timestep is constant.

- I also tried TSROSW. Seems to work quite well in some cases. How does it
compare to ARKIMEX ?

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/c7c681a6/attachment.html>

From asmund.ervik at ntnu.no  Tue Oct  8 07:13:54 2013
From: asmund.ervik at ntnu.no (=?ISO-8859-1?Q?=C5smund_Ervik?=)
Date: Tue, 08 Oct 2013 14:13:54 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <877gdpt6s0.fsf@mcs.anl.gov>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<87d2nht8o2.fsf@mcs.anl.gov> <5252D5E7.8050906@ntnu.no>
	<877gdpt6s0.fsf@mcs.anl.gov>
Message-ID: <5253F702.6080406@ntnu.no>

Hi again,

Regarding the routine DMDAVecGetArrayF90(): I think this warrants it's
own manual page, in addition to the mention on the DMDAVecGetArray()
manpage.

A specific point that confused me is that one should *not* allocate
the pointer array going into this routine. This was far from obvious
from the existing manpage, I only realised it after staring at
dm/ex11f90.F.

?smund


On 07. okt. 2013 17:55, Jed Brown wrote:
> ?smund Ervik <asmund.ervik at ntnu.no> writes:
>> Thanks for this comment; I have been suspecting this is a bad
>> idea, nice to get confirmation. I will follow your advice.
> 
> Cool.
> 
>> Other than that, does the mapping from local to global using
>> Fortran assumed-shape arrays make sense, or is there a better way
>> of doing it?
> 
> It looks okay to me.
> 

From jedbrown at mcs.anl.gov  Tue Oct  8 07:25:00 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 07:25:00 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<87ob70rah5.fsf@mcs.anl.gov>
	<CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>
Message-ID: <87eh7wospv.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> On Tue, Oct 8, 2013 at 12:18 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>> >> - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?
>> >
>> >    Beats me.
>>
>> ARKIMEX should give you a decent integrator with adaptive error control.
>> Use '-ts_arkimex_type 1bee' to use backward Euler with an
>> extrapolation-based error estimator.
>>
>
> Good to know. I tried TSBEULER but it has constant timestep.
>
> -Is there any other TS with adaptive timestep ?

TSARKIMEX (nonlinearly implicit) and TSROSW (linearly implicit) have
embedded error estimators and adaptive controllers.

> -With ARKIMEX, is there a way to control the timestep ? For instance, is it
> possible to control the max factor between two successive timesteps (dt' =
> factor*dt), in order to avoid rejections ?

-ts_adapt_basic_clip <shorten,lengthen> - Admissible decrease/increase in step size
-ts_adapt_basic_safety <safety> - Safety factor relative to target error
-ts_adapt_basic_reject_safety <rsafety> - Extra safety factor to apply if the last step was rejected

> - Is it possible to have Cranck-Nicholson

Crank-Nicolson (spelling)

> with adaptive timestep ? I tried TSCN but it seems timestep is
> constant.

What would you use as an error estimator?  The same approach as 1bee
could be used to write an extrapolation-based method based on
Crank-Nicolson.  Patches welcome if you do this (a simple exercise).

I recommend using an A-stable ARKIMEX method.

> - I also tried TSROSW. Seems to work quite well in some cases. How does it
> compare to ARKIMEX ?

It is linearly implicit.  For problems with sometimes-"stiff"
nonlinearities, ARKIMEX can often take longer time steps.  If ROSW is
taking similar step sizes, it should be more efficient.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/ea29fb88/attachment.pgp>

From pengxwang at hotmail.com  Tue Oct  8 09:26:08 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Tue, 8 Oct 2013 09:26:08 -0500
Subject: [petsc-users] questions about KSP with-pc_type hypre
In-Reply-To: <4DA5639B-F203-4B37-94C9-676716E89378@mcs.anl.gov>
References: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>,
	<53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>
	<BAY179-W843968F8DE0BE9372F27F4B4130@phx.gbl>,
	<4DA5639B-F203-4B37-94C9-676716E89378@mcs.anl.gov>
Message-ID: <BAY179-W8743FCD035C4CBAF70A96AB41C0@phx.gbl>



> Subject: Re: [petsc-users] questions about KSP with-pc_type hypre
> From: bsmith at mcs.anl.gov
> Date: Mon, 7 Oct 2013 16:10:10 -0500
> CC: mfadams at lbl.gov; petsc-users at mcs.anl.gov
> To: pengxwang at hotmail.com
> 
> 
> On Oct 7, 2013, at 3:28 PM, Roc Wang <pengxwang at hotmail.com> wrote:
> 
> > Sorry for the mis-info. 
> > 
> > Yes, I have Dirichlet BCs. The smaller round plate on the left side and the rectangular plane on the right size are the Dirichlet BCs. And the inside the hole of the rectangular plane is Neumann BCs.  I am just wondering the Neumann BCs of the hole have some effect which lead to divergence.  
> 
>    "Some" Neumann boundary conditions shouldn't have much effect. Are you sure the Neumann BCs are applied correctly and that Neumann BCs are not being applied everywhere?  If Neumann BCs are applied almost everywhere on the boundary that can affect convergence.

Yes, the Neumann BCs are applied correctly, since the solution is correct if the KSP solver runs without -pc_type hypre option. On the right side of the domain, the Neumann BCs are only within a hole in the center of the plane.  
> 
>    What happens with -pc_type lu?  Does it solve the linear system and give back answers you expect?

No, it doesn't work if I use -pc_type lu. The error information shows:

[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Matrix format mpiaij does not have a built-in PETSc LU!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 6, Mon Feb 11 12:26:34 CST 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./x.r on a arch-linu named athena.coeaccess.psu.edu by pwang Tue Oct  8 10:23:50 2013
[0]PETSC ERROR: Libraries linked from /home/pwang/soft/petsc-3.3-p6/arch-linux2-c-debug/lib
[0]PETSC ERROR: Configure run at Fri Oct  4 15:18:39 2013
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack --dtime elap.:0     0.0000  1000000  
ownload-mpich --with-cxx=g++ --download-hypre=1 --download-ml=1 --download-hdf5=1
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3876 in src/mat/interface/matrix.c
[0]PETSC ERROR: PCSetUp_LU() line 133 in src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: PCSetUp() line 832 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: LinearSolver() line 264 in "unknowndirectory/"src/solver.cpp
rank 2 in job 959  athena.coeaccess.psu.edu_42121   caused collective abort of all ranks
  exit status of rank 2: killed by signal 6 
rank 1 in job 959  athena.coeaccess.psu.edu_42121   caused collective abort of all ranks
  exit status of rank 1: killed by signal 6 

How can I make -pc_type lu work? Thanks a lot


> 
> 
>    Barry
> 
> > 
> > I will try gamg as your suggestion. Thanks.
> > 
> > 
> > Subject: Re: [petsc-users] questions about KSP with-pc_type hypre
> > From: mfadams at lbl.gov
> > Date: Mon, 7 Oct 2013 16:16:30 -0400
> > CC: petsc-users at mcs.anl.gov
> > To: pengxwang at hotmail.com
> > 
> > I assume this is a Lapacian and its not a pure Neumann problem.  I did not quite follow you description.  Do you have some Dirichlet BCs?
> > 
> > Hypre is a very robust solver, for a scalable solver.  ILU, the default PC, is very robust but slow (that's why its the default).  You could try gamg, another AMG solver to get another data point.
> > 
> > On Oct 7, 2013, at 3:08 PM, Roc Wang <pengxwang at hotmail.com> wrote:
> > 
> > Hello,
> > 
> >    I am trying to solve a linear system with KSP for a 3-D domain.  The computational domain is a cubic space with Neumann condition on the lateral sides( y, and z directions).  A smaller round plate is on the left side in x direction.  The right side is a rectangular plane either with or without a hole.  In testing studies, the KSP solver works fine both with and without -pc_type hypre runtime option for the domain with a whole plane(without a hole) on the right, but diverges for the case of a plane with a hole with -ps_type hypre option. 
> > 
> >   My question is whether the hypre pre-conditioner is sensitive to the geometry boundary?  If the answer is yes, is there any other pre-conditioner available for my particular case(the domain with a plane which with a hole on the right side) to reduce the number of iterations? Thanks in advance.
> > 
> > 
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/0e9c63c6/attachment.html>

From jedbrown at mcs.anl.gov  Tue Oct  8 09:33:29 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 09:33:29 -0500
Subject: [petsc-users] questions about KSP with-pc_type hypre
In-Reply-To: <BAY179-W8743FCD035C4CBAF70A96AB41C0@phx.gbl>
References: <BAY179-W8CAB251183898F34ED140B4130@phx.gbl>
	<53E57562-39C1-4411-A44F-BD275E76A485@lbl.gov>
	<BAY179-W843968F8DE0BE9372F27F4B4130@phx.gbl>
	<4DA5639B-F203-4B37-94C9-676716E89378@mcs.anl.gov>
	<BAY179-W8743FCD035C4CBAF70A96AB41C0@phx.gbl>
Message-ID: <87y563omrq.fsf@mcs.anl.gov>

Roc Wang <pengxwang at hotmail.com> writes:
>>    What happens with -pc_type lu?  Does it solve the linear system and give back answers you expect?
>
> No, it doesn't work if I use -pc_type lu. The error information shows:
>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Matrix format mpiaij does not have a built-in PETSc LU!

If you must run in parallel, either use a third-party solver such as
-pc_type lu -pc_factor_mat_solver_package superlu_dist (configure with
--download-superlu_dist) or use -pc_type redundant (for testing).

I recommend getting your solver working in serial first.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/298692c3/attachment-0001.pgp>

From christophe.ortiz at ciemat.es  Tue Oct  8 09:34:45 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 8 Oct 2013 16:34:45 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87eh7wospv.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<87ob70rah5.fsf@mcs.anl.gov>
	<CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>
	<87eh7wospv.fsf@mcs.anl.gov>
Message-ID: <CANBrw+7Nu10EC0KB7tzV32v_nzySqZ_5vDKTBoxBUVVh65p7Sg@mail.gmail.com>

On Tue, Oct 8, 2013 at 2:25 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > On Tue, Oct 8, 2013 at 12:18 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> >
> >> Barry Smith <bsmith at mcs.anl.gov> writes:
> >> >> - In this type of problem, which TS scheme is recommended ?
> TSARKIMEX ?
> >> >
> >> >    Beats me.
> >>
> >> ARKIMEX should give you a decent integrator with adaptive error control.
> >> Use '-ts_arkimex_type 1bee' to use backward Euler with an
> >> extrapolation-based error estimator.
> >>
> >
> > Good to know. I tried TSBEULER but it has constant timestep.
> >
> > -Is there any other TS with adaptive timestep ?
>
> TSARKIMEX (nonlinearly implicit) and TSROSW (linearly implicit) have
> embedded error estimators and adaptive controllers.
>
> > -With ARKIMEX, is there a way to control the timestep ? For instance, is
> it
> > possible to control the max factor between two successive timesteps (dt'
> =
> > factor*dt), in order to avoid rejections ?
>
> -ts_adapt_basic_clip <shorten,lengthen> - Admissible decrease/increase in
> step size
> -ts_adapt_basic_safety <safety> - Safety factor relative to target error
> -ts_adapt_basic_reject_safety <rsafety> - Extra safety factor to apply if
> the last step was rejected
>


What are the names of the subroutines to access these options directly
within the code ? I see there is TSAdaptSetStepLimits but this is to set
the min and max timestep size.


>
> > - Is it possible to have Cranck-Nicholson
>
> Crank-Nicolson (spelling)
>
> > with adaptive timestep ? I tried TSCN but it seems timestep is
> > constant.
>
> What would you use as an error estimator?  The same approach as 1bee
> could be used to write an extrapolation-based method based on
> Crank-Nicolson.  Patches welcome if you do this (a simple exercise).
>
> I recommend using an A-stable ARKIMEX method.
>

Ok, thanks.


>
> > - I also tried TSROSW. Seems to work quite well in some cases. How does
> it
> > compare to ARKIMEX ?
>
> It is linearly implicit.  For problems with sometimes-"stiff"
> nonlinearities, ARKIMEX can often take longer time steps.  If ROSW is
> taking similar step sizes, it should be more efficient.
>

Good to know.

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/22887abb/attachment.html>

From jedbrown at mcs.anl.gov  Tue Oct  8 09:55:26 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 09:55:26 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7Nu10EC0KB7tzV32v_nzySqZ_5vDKTBoxBUVVh65p7Sg@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<87ob70rah5.fsf@mcs.anl.gov>
	<CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>
	<87eh7wospv.fsf@mcs.anl.gov>
	<CANBrw+7Nu10EC0KB7tzV32v_nzySqZ_5vDKTBoxBUVVh65p7Sg@mail.gmail.com>
Message-ID: <87vc17olr5.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> What are the names of the subroutines to access these options directly
> within the code ? I see there is TSAdaptSetStepLimits but this is to set
> the min and max timestep size.

Sorry, they are not exposed at this time.  They can (and should) be
added, but you can also register your own controller.  If you'd like to
help out, you can prepare a patch that adds TSAdaptBasicSetClip() and
similar functions to src/ts/adapt/impls/basic/adaptbasic.c.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/2d1710c5/attachment.pgp>

From hariseldon99 at gmail.com  Tue Oct  8 13:27:06 2013
From: hariseldon99 at gmail.com (Analabha Roy)
Date: Tue, 8 Oct 2013 23:57:06 +0530
Subject: [petsc-users] Get a smaller vector with PETSc
In-Reply-To: <1576B5B5-73F9-4604-9708-CC3634005607@mcs.anl.gov>
References: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>
	<1576B5B5-73F9-4604-9708-CC3634005607@mcs.anl.gov>
Message-ID: <CAKiGokHENCYQQenEX_Hdi6n8O0-A=ZcPeek5s+Vp7HZ2=tOYJw@mail.gmail.com>

On Sun, Oct 6, 2013 at 7:29 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   VecScatterCreate().
>


What about VecGetSubVector()<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html>?



>
>    If you wish the sub vector to live on a single process then create a
> sequential vector of the appropriate size on that one process and
> sequential vectors of size zero on all other processes and scatter into
> them.
>
>    Barry
>
> On Oct 2, 2013, at 6:27 AM, Rui Emanuel Ferreira da Silva <
> rui.silva at uam.es> wrote:
>
> > I am writing to you to ask some technical problems that I am dealing
> > with the use of PETSc.
> >
> > The problem that I am facing is the following: I have defined a vector
> in Petsc which was created with VecCreateMPI. After that, I read this
> vector from an external file with a PETSc viewer.
> >
> > So if the initial vector is V with size N, I want to obtain a smaller
> vector U with size M, where M<N. I have an array of integers that are the
> indexes of V that I want to pass to U.
> >
> > How can I do this?
> >
> > Regards
> >
> > -------------------
> > Rui Silva
> > EMTCCM (European Master in Theoretical Chemistry and Computational
> Modelling)
> > UAM, Departamento de Qu?mica, M?dulo 13
> > CAMPUS
> http://www.uam.es/departamentos/ciencias/quimica/spline/index.html
> > -------------------
> >
>
>


-- 
---
*Analabha Roy*
C.S.I.R <http://www.csir.res.in>  Senior Research
Associate<http://csirhrdg.res.in/poolsra.htm>
Saha Institute of Nuclear Physics <http://www.saha.ac.in>
Section 1, Block AF
Bidhannagar, Calcutta 700064
India
*Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
*Webpage*: http://www.ph.utexas.edu/~daneel/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/aa25111d/attachment.html>

From knepley at gmail.com  Tue Oct  8 13:35:05 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Oct 2013 13:35:05 -0500
Subject: [petsc-users] Get a smaller vector with PETSc
In-Reply-To: <CAKiGokHENCYQQenEX_Hdi6n8O0-A=ZcPeek5s+Vp7HZ2=tOYJw@mail.gmail.com>
References: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>
	<1576B5B5-73F9-4604-9708-CC3634005607@mcs.anl.gov>
	<CAKiGokHENCYQQenEX_Hdi6n8O0-A=ZcPeek5s+Vp7HZ2=tOYJw@mail.gmail.com>
Message-ID: <CAMYG4G=U1Ca1PdDXTgx=BWtUk74NYtBkAgiaOUgxra0Lth3UhQ@mail.gmail.com>

On Tue, Oct 8, 2013 at 1:27 PM, Analabha Roy <hariseldon99 at gmail.com> wrote:

>
>
>
> On Sun, Oct 6, 2013 at 7:29 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   VecScatterCreate().
>>
>
>
> What about VecGetSubVector()<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html>?
>

Only works for local subsets.

  Matt


>
>
>>
>>    If you wish the sub vector to live on a single process then create a
>> sequential vector of the appropriate size on that one process and
>> sequential vectors of size zero on all other processes and scatter into
>> them.
>>
>>    Barry
>>
>> On Oct 2, 2013, at 6:27 AM, Rui Emanuel Ferreira da Silva <
>> rui.silva at uam.es> wrote:
>>
>> > I am writing to you to ask some technical problems that I am dealing
>> > with the use of PETSc.
>> >
>> > The problem that I am facing is the following: I have defined a vector
>> in Petsc which was created with VecCreateMPI. After that, I read this
>> vector from an external file with a PETSc viewer.
>> >
>> > So if the initial vector is V with size N, I want to obtain a smaller
>> vector U with size M, where M<N. I have an array of integers that are the
>> indexes of V that I want to pass to U.
>> >
>> > How can I do this?
>> >
>> > Regards
>> >
>> > -------------------
>> > Rui Silva
>> > EMTCCM (European Master in Theoretical Chemistry and Computational
>> Modelling)
>> > UAM, Departamento de Qu?mica, M?dulo 13
>> > CAMPUS
>> http://www.uam.es/departamentos/ciencias/quimica/spline/index.html
>> > -------------------
>> >
>>
>>
>
>
> --
> ---
> *Analabha Roy*
> C.S.I.R <http://www.csir.res.in>  Senior Research Associate<http://csirhrdg.res.in/poolsra.htm>
> Saha Institute of Nuclear Physics <http://www.saha.ac.in>
> Section 1, Block AF
> Bidhannagar, Calcutta 700064
> India
> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
> *Webpage*: http://www.ph.utexas.edu/~daneel/
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/6c35da6f/attachment.html>

From sonyablade2010 at hotmail.com  Tue Oct  8 13:39:39 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Tue, 8 Oct 2013 19:39:39 +0100
Subject: [petsc-users] SlepcInitialize Segmentation Fault
Message-ID: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>

DEar All,

I'm receiving the segmentation fault during the program run and this happens exactly at the?
SlepcInitialize function and I've no idea on what can be wrong, the error messages produced?
from compiler is as shown below. Any clue on what is wrong?


ORIGINAL CODE SNIPPET?

int main( int argc, char **argv )
{
? Vec ? ? ? ? ? ?v0,vec,vec1,*evec,gamma,vec_test,result_vec; ? ? ? ? ? ? ?/* initial vector */
? Mat ? ? ? ? ? ?B, C, K, M, eigen_mat; ? ? ? ? ? ? ? /* operator matrix */
? EPS ? ? ? ? ? ?eps; ? ? ? ? ? ? /* eigenproblem solver context */
? const EPSType ?type;
? PetscInt ? ? ? N,m=15,nev,max_size,row,column,test_int,g_row,g_col;
? PetscErrorCode ierr;
? PetscScalar eigen_r,eigen_i,test_scalar, *vector_array,*vec1arr;
? PetscReal norm;
? double value,test;
? int i,j, counter, counter2;
? FILE *myfile;

SlepcInitialize(&argc,&argv,NULL,NULL);



ERROR MESSAGES?

At D:\TEST_FOLDER_asus\PROJECTS\CBFortran\Slepc_C\Slepc_Subspace\main.c:24
Program received signal SIGSEGV, Segmentation fault.
In strncat () (C:\Windows\SysWOW64\msvcrt.dll)
217	/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c: No such file or directory.
#2 ?0x0065f761 in PetscSetProgramName (name=0x7370652d <Address 0x7370652d out of bounds>) at /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:217
Debugger finished with status 0 		 	   		  

From knepley at gmail.com  Tue Oct  8 13:53:37 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Oct 2013 13:53:37 -0500
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
Message-ID: <CAMYG4GmyJs8ENhqHttTbe8R_it-huAn=3_-4+D+2r6yv6chzVQ@mail.gmail.com>

On Tue, Oct 8, 2013 at 1:39 PM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> DEar All,
>
> I'm receiving the segmentation fault during the program run and this
> happens exactly at the
> SlepcInitialize function and I've no idea on what can be wrong, the error
> messages produced
> from compiler is as shown below. Any clue on what is wrong?
>

Somehow the first argument is bogus instead of being the program name. Is
this a C or Fortran
program?

  Matt


> ORIGINAL CODE SNIPPET
>
> int main( int argc, char **argv )
> {
>   Vec            v0,vec,vec1,*evec,gamma,vec_test,result_vec;
>  /* initial vector */
>   Mat            B, C, K, M, eigen_mat;               /* operator matrix */
>   EPS            eps;             /* eigenproblem solver context */
>   const EPSType  type;
>   PetscInt       N,m=15,nev,max_size,row,column,test_int,g_row,g_col;
>   PetscErrorCode ierr;
>   PetscScalar eigen_r,eigen_i,test_scalar, *vector_array,*vec1arr;
>   PetscReal norm;
>   double value,test;
>   int i,j, counter, counter2;
>   FILE *myfile;
>
> SlepcInitialize(&argc,&argv,NULL,NULL);
>
>
>
> ERROR MESSAGES
>
> At D:\TEST_FOLDER_asus\PROJECTS\CBFortran\Slepc_C\Slepc_Subspace\main.c:24
> Program received signal SIGSEGV, Segmentation fault.
> In strncat () (C:\Windows\SysWOW64\msvcrt.dll)
> 217
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c: No such
> file or directory.
> #2  0x0065f761 in PetscSetProgramName (name=0x7370652d <Address 0x7370652d
> out of bounds>) at
> /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:217
> Debugger finished with status 0




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/e3144874/attachment.html>

From sonyablade2010 at hotmail.com  Tue Oct  8 14:07:45 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Tue, 8 Oct 2013 20:07:45 +0100
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <CAMYG4GmyJs8ENhqHttTbe8R_it-huAn=3_-4+D+2r6yv6chzVQ@mail.gmail.com>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>,
	<CAMYG4GmyJs8ENhqHttTbe8R_it-huAn=3_-4+D+2r6yv6chzVQ@mail.gmail.com>
Message-ID: <DUB119-W241CE87C63E3545A5A4DD4C81C0@phx.gbl>

> Somehow the first argument is bogus instead of being the program name. 
> Is this a C or Fortran 
> program? 
> 
> Matt 
> 

C Program 		 	   		  

From knepley at gmail.com  Tue Oct  8 14:08:27 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Oct 2013 14:08:27 -0500
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W241CE87C63E3545A5A4DD4C81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
	<CAMYG4GmyJs8ENhqHttTbe8R_it-huAn=3_-4+D+2r6yv6chzVQ@mail.gmail.com>
	<DUB119-W241CE87C63E3545A5A4DD4C81C0@phx.gbl>
Message-ID: <CAMYG4G=s3+v5JZ-o3Y7NO0pAoL1aqxRcBoTxT-L3JQOtNcWxMw@mail.gmail.com>

On Tue, Oct 8, 2013 at 2:07 PM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> > Somehow the first argument is bogus instead of being the program name.
> > Is this a C or Fortran
> > program?
> >
> > Matt
> >
>
> C Program


Run with -start_in_debugger and get a stack trace using 'where'.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/2dbd6311/attachment-0001.html>

From balay at mcs.anl.gov  Tue Oct  8 14:08:51 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 8 Oct 2013 14:08:51 -0500 (CDT)
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
Message-ID: <alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>

Do you get this error with petsc examples or slepc examples?

Satish

On Tue, 8 Oct 2013, Sonya Blade wrote:

> DEar All,
> 
> I'm receiving the segmentation fault during the program run and this happens exactly at the?
> SlepcInitialize function and I've no idea on what can be wrong, the error messages produced?
> from compiler is as shown below. Any clue on what is wrong?
> 
> 
> ORIGINAL CODE SNIPPET?
> 
> int main( int argc, char **argv )
> {
> ? Vec ? ? ? ? ? ?v0,vec,vec1,*evec,gamma,vec_test,result_vec; ? ? ? ? ? ? ?/* initial vector */
> ? Mat ? ? ? ? ? ?B, C, K, M, eigen_mat; ? ? ? ? ? ? ? /* operator matrix */
> ? EPS ? ? ? ? ? ?eps; ? ? ? ? ? ? /* eigenproblem solver context */
> ? const EPSType ?type;
> ? PetscInt ? ? ? N,m=15,nev,max_size,row,column,test_int,g_row,g_col;
> ? PetscErrorCode ierr;
> ? PetscScalar eigen_r,eigen_i,test_scalar, *vector_array,*vec1arr;
> ? PetscReal norm;
> ? double value,test;
> ? int i,j, counter, counter2;
> ? FILE *myfile;
> 
> SlepcInitialize(&argc,&argv,NULL,NULL);
> 
> 
> 
> ERROR MESSAGES?
> 
> At D:\TEST_FOLDER_asus\PROJECTS\CBFortran\Slepc_C\Slepc_Subspace\main.c:24
> Program received signal SIGSEGV, Segmentation fault.
> In strncat () (C:\Windows\SysWOW64\msvcrt.dll)
> 217	/cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c: No such file or directory.
> #2 ?0x0065f761 in PetscSetProgramName (name=0x7370652d <Address 0x7370652d out of bounds>) at /cygdrive/d/TEST_FOLDER_asus/petsc-3.4.2/src/sys/objects/options.c:217
> Debugger finished with status 0 		 	   		  

From sonyablade2010 at hotmail.com  Tue Oct  8 14:17:31 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Tue, 8 Oct 2013 20:17:31 +0100
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>,
	<alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>
Message-ID: <DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>

> Do you get this error with petsc examples or slepc examples?
>
> Satish
>

I get this from slepc example, to be specific modified Ex2 to suit my needs,?
I tested it for Petsc HelloWorld example which ran w/o segfault. But I'm not?
sure whether all the funcitonality of Petsc is being addressed ?with runing such simple?
example, in any case PetscInitialize didn't raise segfault. 		 	   		  

From knepley at gmail.com  Tue Oct  8 14:19:19 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Oct 2013 14:19:19 -0500
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
	<alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>
	<DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>
Message-ID: <CAMYG4GkTTd4h-4tgU8s6044ikek-ta4Pr_NWj4_OBpiEEvLHgQ@mail.gmail.com>

On Tue, Oct 8, 2013 at 2:17 PM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> > Do you get this error with petsc examples or slepc examples?
> >
> > Satish
> >
>
> I get this from slepc example, to be specific modified Ex2 to suit my
> needs,
> I tested it for Petsc HelloWorld example which ran w/o segfault. But I'm
> not
> sure whether all the funcitonality of Petsc is being addressed  with
> runing such simple
> example, in any case PetscInitialize didn't raise segfault.
>


Definitely this is being exercised, so there is something in your
modification. Again, look in the debugger.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/275c1605/attachment.html>

From balay at mcs.anl.gov  Tue Oct  8 14:28:12 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 8 Oct 2013 14:28:12 -0500 (CDT)
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>,
	<alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>
	<DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>
Message-ID: <alpine.LFD.2.03.1310081427130.2030@mcs.anl.gov>

On Tue, 8 Oct 2013, Sonya Blade wrote:

> > Do you get this error with petsc examples or slepc examples?
> >
> > Satish
> >
> 
> I get this from slepc example, to be specific modified Ex2 to suit my needs,?

But do you get this error with 'unmodified' slepc examples? [with slepc makefile?]

Satish

> I tested it for Petsc HelloWorld example which ran w/o segfault. But I'm not?
> sure whether all the funcitonality of Petsc is being addressed ?with runing such simple?
> example, in any case PetscInitialize didn't raise segfault. 		 	   		  

From sonyablade2010 at hotmail.com  Tue Oct  8 14:44:09 2013
From: sonyablade2010 at hotmail.com (Sonya Blade)
Date: Tue, 8 Oct 2013 20:44:09 +0100
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <CAMYG4GkTTd4h-4tgU8s6044ikek-ta4Pr_NWj4_OBpiEEvLHgQ@mail.gmail.com>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>,
	<alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>,
	<DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>,
	<CAMYG4GkTTd4h-4tgU8s6044ikek-ta4Pr_NWj4_OBpiEEvLHgQ@mail.gmail.com>
Message-ID: <DUB119-W34A4343B363C60BEA9CB7DC81C0@phx.gbl>

>But do you get this error with 'unmodified' slepc examples? [with slepc makefile?]
>
>Satish

Yes, and call stack points out to PetscErrorPrintfInitialize()



> Definitely this is being exercised, so there is something in your
> modification. Again, look in the debugger.
>
> Matt

call stack points out to PetscErrorPrintfInitialize(). ?

I'd like also to note that I'm using CodeBlocks on Win8 if that gives any hint. 		 	   		  

From jedbrown at mcs.anl.gov  Tue Oct  8 14:55:01 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 14:55:01 -0500
Subject: [petsc-users] Get a smaller vector with PETSc
In-Reply-To: <CAMYG4G=U1Ca1PdDXTgx=BWtUk74NYtBkAgiaOUgxra0Lth3UhQ@mail.gmail.com>
References: <20131002132750.Horde.lmvLJOJsT11STAM2u6cF2-A@webmail.uam.es>
	<1576B5B5-73F9-4604-9708-CC3634005607@mcs.anl.gov>
	<CAKiGokHENCYQQenEX_Hdi6n8O0-A=ZcPeek5s+Vp7HZ2=tOYJw@mail.gmail.com>
	<CAMYG4G=U1Ca1PdDXTgx=BWtUk74NYtBkAgiaOUgxra0Lth3UhQ@mail.gmail.com>
Message-ID: <877gdnmtbe.fsf@mcs.anl.gov>

Matthew Knepley <knepley at gmail.com> writes:
>> What about VecGetSubVector()<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html>?
>>
>
> Only works for local subsets.

Actually, VecGetSubVector() should work since in the case where the IS
is not both local and contiguous, it creates a VecScatter internally.
If you want a separate vector that you can keep (don't have to
VecRestoreSubVector) then you should use VecScatter yourself.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/9bc6fc72/attachment.pgp>

From knepley at gmail.com  Tue Oct  8 15:05:41 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 Oct 2013 15:05:41 -0500
Subject: [petsc-users] SlepcInitialize Segmentation Fault
In-Reply-To: <DUB119-W34A4343B363C60BEA9CB7DC81C0@phx.gbl>
References: <DUB119-W2376ACA256F3B88A09140EC81C0@phx.gbl>
	<alpine.LFD.2.03.1310081408350.2030@mcs.anl.gov>
	<DUB119-W20C88110B828A4BB0C1979C81C0@phx.gbl>
	<CAMYG4GkTTd4h-4tgU8s6044ikek-ta4Pr_NWj4_OBpiEEvLHgQ@mail.gmail.com>
	<DUB119-W34A4343B363C60BEA9CB7DC81C0@phx.gbl>
Message-ID: <CAMYG4GmNzXtfDYjG93i4QzDy4wW3spgsnUEjRbNJw+y7Uv2L8A@mail.gmail.com>

On Tue, Oct 8, 2013 at 2:44 PM, Sonya Blade <sonyablade2010 at hotmail.com>wrote:

> >But do you get this error with 'unmodified' slepc examples? [with slepc
> makefile?]
> >
> >Satish
>
> Yes, and call stack points out to PetscErrorPrintfInitialize()
>
>
>
> > Definitely this is being exercised, so there is something in your
> > modification. Again, look in the debugger.
> >
> > Matt
>
> call stack points out to PetscErrorPrintfInitialize().
>

Show me.

  Matt


> I'd like also to note that I'm using CodeBlocks on Win8 if that gives any
> hint.




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/0d149517/attachment.html>

From jolivet at ann.jussieu.fr  Tue Oct  8 16:11:31 2013
From: jolivet at ann.jussieu.fr (Pierre Jolivet)
Date: Tue, 8 Oct 2013 23:11:31 +0200 (CEST)
Subject: [petsc-users] Problem with AMG packages
Message-ID: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>

Dear all,
I'm trying to compare linear solvers for a simple Poisson equation in 3D.
I thought that MG was the way to go, but looking at my log, the
performance looks abysmal (I know that the matrices are way too small but
if I go bigger, it just never performs a single iteration ..). Even though
this is neither the BoomerAMG nor the ML mailing list, could you please
tell me if PETSc sets some default flags that make the setup for those
solvers so slow for this simple problem ? The performance of (G)ASM is in
comparison much better.

Thanks in advance for your help.

PS: first the BoomerAMG log, then ML (much more verbose, sorry).

  0 KSP Residual norm 1.599647112604e+00
  1 KSP Residual norm 5.450838232404e-02
  2 KSP Residual norm 3.549673478318e-03
  3 KSP Residual norm 2.901826808841e-04
  4 KSP Residual norm 2.574235778729e-05
  5 KSP Residual norm 2.253410171682e-06
  6 KSP Residual norm 1.871067784877e-07
  7 KSP Residual norm 1.681162800670e-08
  8 KSP Residual norm 2.120841512414e-09
KSP Object: 2048 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=200, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 2048 MPI processes
  type: hypre
    HYPRE BoomerAMG preconditioning
    HYPRE BoomerAMG: Cycle type V
    HYPRE BoomerAMG: Maximum number of levels 25
    HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
    HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
    HYPRE BoomerAMG: Threshold for strong coupling 0.25
    HYPRE BoomerAMG: Interpolation truncation factor 0
    HYPRE BoomerAMG: Interpolation: max elements per row 0
    HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
    HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
    HYPRE BoomerAMG: Maximum row sums 0.9
    HYPRE BoomerAMG: Sweeps down         1
    HYPRE BoomerAMG: Sweeps up           1
    HYPRE BoomerAMG: Sweeps on coarse    1
    HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
    HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
    HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
    HYPRE BoomerAMG: Relax weight  (all)      1
    HYPRE BoomerAMG: Outer relax weight (all) 1
    HYPRE BoomerAMG: Using CF-relaxation
    HYPRE BoomerAMG: Measure type        local
    HYPRE BoomerAMG: Coarsen type        Falgout
    HYPRE BoomerAMG: Interpolation type  classical
  linear system matrix = precond matrix:
  Matrix Object:   2048 MPI processes
    type: mpiaij
    rows=4173281, cols=4173281
    total: nonzeros=102576661, allocated nonzeros=102576661
    total number of mallocs used during MatSetValues calls =0
      not using I-node (on process 0) routines
 --- system solved with PETSc (in 1.005199e+02 seconds)

  0 KSP Residual norm 2.368804472986e-01
  1 KSP Residual norm 5.676430019132e-02
  2 KSP Residual norm 1.898005876002e-02
  3 KSP Residual norm 6.193922902926e-03
  4 KSP Residual norm 2.008448794493e-03
  5 KSP Residual norm 6.390465670228e-04
  6 KSP Residual norm 2.157709394389e-04
  7 KSP Residual norm 7.295973819979e-05
  8 KSP Residual norm 2.358343271482e-05
  9 KSP Residual norm 7.489696222066e-06
 10 KSP Residual norm 2.390946857593e-06
 11 KSP Residual norm 8.068086385140e-07
 12 KSP Residual norm 2.706607789749e-07
 13 KSP Residual norm 8.636910863376e-08
 14 KSP Residual norm 2.761981175852e-08
 15 KSP Residual norm 8.755459874369e-09
 16 KSP Residual norm 2.708848598341e-09
 17 KSP Residual norm 8.968748876265e-10
KSP Object: 2048 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=200, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 2048 MPI processes
  type: ml
    MG: type is MULTIPLICATIVE, levels=3 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     2048 MPI processes
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     2048 MPI processes
      type: redundant
        Redundant preconditioner: First (color=0) of 2048 PCs follows
      KSP Object:      (mg_coarse_redundant_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_coarse_redundant_)       1 MPI processes
        type: lu
          LU: out-of-place factorization
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot
          matrix ordering: nd
          factor fill ratio given 5, needed 4.38504
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=2055, cols=2055
                package used to perform factorization: petsc
                total: nonzeros=2476747, allocated nonzeros=2476747
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 1638 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=2055, cols=2055
          total: nonzeros=564817, allocated nonzeros=1093260
          total number of mallocs used during MatSetValues calls =0
            not using I-node routines
      linear system matrix = precond matrix:
      Matrix Object:       2048 MPI processes
        type: mpiaij
        rows=2055, cols=2055
        total: nonzeros=564817, allocated nonzeros=564817
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     2048 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_1_)     2048 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       2048 MPI processes
        type: mpiaij
        rows=30194, cols=30194
        total: nonzeros=3368414, allocated nonzeros=3368414
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     2048 MPI processes
      type: richardson
        Richardson: damping factor=1
      maximum iterations=2
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using NONE norm type for convergence test
    PC Object:    (mg_levels_2_)     2048 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       2048 MPI processes
        type: mpiaij
        rows=531441, cols=531441
        total: nonzeros=12476324, allocated nonzeros=12476324
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   2048 MPI processes
    type: mpiaij
    rows=531441, cols=531441
    total: nonzeros=12476324, allocated nonzeros=12476324
    total number of mallocs used during MatSetValues calls =0
      not using I-node (on process 0) routines
 --- system solved with PETSc (in 2.407844e+02 seconds)



From bsmith at mcs.anl.gov  Tue Oct  8 16:16:55 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 Oct 2013 16:16:55 -0500
Subject: [petsc-users] Problem with AMG packages
In-Reply-To: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>
References: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>
Message-ID: <574A2DCB-6F7B-4279-A4AE-A2A520EE7952@mcs.anl.gov>


  We need the output from running with -log_summary -pc_mg_log

   Also you can run with PETSc's AMG called GAMG (run with -pc_type gamg) This will give the most useful information about where it is spending the time.


   Barry


On Oct 8, 2013, at 4:11 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:

> Dear all,
> I'm trying to compare linear solvers for a simple Poisson equation in 3D.
> I thought that MG was the way to go, but looking at my log, the
> performance looks abysmal (I know that the matrices are way too small but
> if I go bigger, it just never performs a single iteration ..). Even though
> this is neither the BoomerAMG nor the ML mailing list, could you please
> tell me if PETSc sets some default flags that make the setup for those
> solvers so slow for this simple problem ? The performance of (G)ASM is in
> comparison much better.
> 
> Thanks in advance for your help.
> 
> PS: first the BoomerAMG log, then ML (much more verbose, sorry).
> 
>  0 KSP Residual norm 1.599647112604e+00
>  1 KSP Residual norm 5.450838232404e-02
>  2 KSP Residual norm 3.549673478318e-03
>  3 KSP Residual norm 2.901826808841e-04
>  4 KSP Residual norm 2.574235778729e-05
>  5 KSP Residual norm 2.253410171682e-06
>  6 KSP Residual norm 1.871067784877e-07
>  7 KSP Residual norm 1.681162800670e-08
>  8 KSP Residual norm 2.120841512414e-09
> KSP Object: 2048 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=200, initial guess is zero
>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 2048 MPI processes
>  type: hypre
>    HYPRE BoomerAMG preconditioning
>    HYPRE BoomerAMG: Cycle type V
>    HYPRE BoomerAMG: Maximum number of levels 25
>    HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>    HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>    HYPRE BoomerAMG: Threshold for strong coupling 0.25
>    HYPRE BoomerAMG: Interpolation truncation factor 0
>    HYPRE BoomerAMG: Interpolation: max elements per row 0
>    HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>    HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>    HYPRE BoomerAMG: Maximum row sums 0.9
>    HYPRE BoomerAMG: Sweeps down         1
>    HYPRE BoomerAMG: Sweeps up           1
>    HYPRE BoomerAMG: Sweeps on coarse    1
>    HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>    HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>    HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>    HYPRE BoomerAMG: Relax weight  (all)      1
>    HYPRE BoomerAMG: Outer relax weight (all) 1
>    HYPRE BoomerAMG: Using CF-relaxation
>    HYPRE BoomerAMG: Measure type        local
>    HYPRE BoomerAMG: Coarsen type        Falgout
>    HYPRE BoomerAMG: Interpolation type  classical
>  linear system matrix = precond matrix:
>  Matrix Object:   2048 MPI processes
>    type: mpiaij
>    rows=4173281, cols=4173281
>    total: nonzeros=102576661, allocated nonzeros=102576661
>    total number of mallocs used during MatSetValues calls =0
>      not using I-node (on process 0) routines
> --- system solved with PETSc (in 1.005199e+02 seconds)
> 
>  0 KSP Residual norm 2.368804472986e-01
>  1 KSP Residual norm 5.676430019132e-02
>  2 KSP Residual norm 1.898005876002e-02
>  3 KSP Residual norm 6.193922902926e-03
>  4 KSP Residual norm 2.008448794493e-03
>  5 KSP Residual norm 6.390465670228e-04
>  6 KSP Residual norm 2.157709394389e-04
>  7 KSP Residual norm 7.295973819979e-05
>  8 KSP Residual norm 2.358343271482e-05
>  9 KSP Residual norm 7.489696222066e-06
> 10 KSP Residual norm 2.390946857593e-06
> 11 KSP Residual norm 8.068086385140e-07
> 12 KSP Residual norm 2.706607789749e-07
> 13 KSP Residual norm 8.636910863376e-08
> 14 KSP Residual norm 2.761981175852e-08
> 15 KSP Residual norm 8.755459874369e-09
> 16 KSP Residual norm 2.708848598341e-09
> 17 KSP Residual norm 8.968748876265e-10
> KSP Object: 2048 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=200, initial guess is zero
>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 2048 MPI processes
>  type: ml
>    MG: type is MULTIPLICATIVE, levels=3 cycles=v
>      Cycles per PCApply=1
>      Using Galerkin computed coarse grid matrices
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_coarse_)     2048 MPI processes
>      type: preonly
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object:    (mg_coarse_)     2048 MPI processes
>      type: redundant
>        Redundant preconditioner: First (color=0) of 2048 PCs follows
>      KSP Object:      (mg_coarse_redundant_)       1 MPI processes
>        type: preonly
>        maximum iterations=10000, initial guess is zero
>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>        left preconditioning
>        using NONE norm type for convergence test
>      PC Object:      (mg_coarse_redundant_)       1 MPI processes
>        type: lu
>          LU: out-of-place factorization
>          tolerance for zero pivot 2.22045e-14
>          using diagonal shift on blocks to prevent zero pivot
>          matrix ordering: nd
>          factor fill ratio given 5, needed 4.38504
>            Factored matrix follows:
>              Matrix Object:               1 MPI processes
>                type: seqaij
>                rows=2055, cols=2055
>                package used to perform factorization: petsc
>                total: nonzeros=2476747, allocated nonzeros=2476747
>                total number of mallocs used during MatSetValues calls =0
>                  using I-node routines: found 1638 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Matrix Object:         1 MPI processes
>          type: seqaij
>          rows=2055, cols=2055
>          total: nonzeros=564817, allocated nonzeros=1093260
>          total number of mallocs used during MatSetValues calls =0
>            not using I-node routines
>      linear system matrix = precond matrix:
>      Matrix Object:       2048 MPI processes
>        type: mpiaij
>        rows=2055, cols=2055
>        total: nonzeros=564817, allocated nonzeros=564817
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node (on process 0) routines
>  Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object:    (mg_levels_1_)     2048 MPI processes
>      type: richardson
>        Richardson: damping factor=1
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_1_)     2048 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Matrix Object:       2048 MPI processes
>        type: mpiaij
>        rows=30194, cols=30194
>        total: nonzeros=3368414, allocated nonzeros=3368414
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node (on process 0) routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 2 -------------------------------
>    KSP Object:    (mg_levels_2_)     2048 MPI processes
>      type: richardson
>        Richardson: damping factor=1
>      maximum iterations=2
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using nonzero initial guess
>      using NONE norm type for convergence test
>    PC Object:    (mg_levels_2_)     2048 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1
>      linear system matrix = precond matrix:
>      Matrix Object:       2048 MPI processes
>        type: mpiaij
>        rows=531441, cols=531441
>        total: nonzeros=12476324, allocated nonzeros=12476324
>        total number of mallocs used during MatSetValues calls =0
>          not using I-node (on process 0) routines
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  linear system matrix = precond matrix:
>  Matrix Object:   2048 MPI processes
>    type: mpiaij
>    rows=531441, cols=531441
>    total: nonzeros=12476324, allocated nonzeros=12476324
>    total number of mallocs used during MatSetValues calls =0
>      not using I-node (on process 0) routines
> --- system solved with PETSc (in 2.407844e+02 seconds)
> 
> 


From jolivet at ann.jussieu.fr  Tue Oct  8 16:46:29 2013
From: jolivet at ann.jussieu.fr (Pierre Jolivet)
Date: Tue, 8 Oct 2013 23:46:29 +0200 (CEST)
Subject: [petsc-users] Problem with AMG packages
In-Reply-To: <574A2DCB-6F7B-4279-A4AE-A2A520EE7952@mcs.anl.gov>
References: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>
	<574A2DCB-6F7B-4279-A4AE-A2A520EE7952@mcs.anl.gov>
Message-ID: <50380.82.224.94.31.1381268789.squirrel@webmail.ann.jussieu.fr>

Please find the log for BoomerAMG, ML and GAMG attached. The set up for
GAMG doesn't look so bad compared to the other packages, so I'm wondering
what is going on with those ?

>
>   We need the output from running with -log_summary -pc_mg_log
>
>    Also you can run with PETSc's AMG called GAMG (run with -pc_type gamg)
> This will give the most useful information about where it is spending
> the time.
>
>
>    Barry
>
>
> On Oct 8, 2013, at 4:11 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:
>
>> Dear all,
>> I'm trying to compare linear solvers for a simple Poisson equation in
>> 3D.
>> I thought that MG was the way to go, but looking at my log, the
>> performance looks abysmal (I know that the matrices are way too small
>> but
>> if I go bigger, it just never performs a single iteration ..). Even
>> though
>> this is neither the BoomerAMG nor the ML mailing list, could you please
>> tell me if PETSc sets some default flags that make the setup for those
>> solvers so slow for this simple problem ? The performance of (G)ASM is
>> in
>> comparison much better.
>>
>> Thanks in advance for your help.
>>
>> PS: first the BoomerAMG log, then ML (much more verbose, sorry).
>>
>>  0 KSP Residual norm 1.599647112604e+00
>>  1 KSP Residual norm 5.450838232404e-02
>>  2 KSP Residual norm 3.549673478318e-03
>>  3 KSP Residual norm 2.901826808841e-04
>>  4 KSP Residual norm 2.574235778729e-05
>>  5 KSP Residual norm 2.253410171682e-06
>>  6 KSP Residual norm 1.871067784877e-07
>>  7 KSP Residual norm 1.681162800670e-08
>>  8 KSP Residual norm 2.120841512414e-09
>> KSP Object: 2048 MPI processes
>>  type: gmres
>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>    GMRES: happy breakdown tolerance 1e-30
>>  maximum iterations=200, initial guess is zero
>>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 2048 MPI processes
>>  type: hypre
>>    HYPRE BoomerAMG preconditioning
>>    HYPRE BoomerAMG: Cycle type V
>>    HYPRE BoomerAMG: Maximum number of levels 25
>>    HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>>    HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>>    HYPRE BoomerAMG: Threshold for strong coupling 0.25
>>    HYPRE BoomerAMG: Interpolation truncation factor 0
>>    HYPRE BoomerAMG: Interpolation: max elements per row 0
>>    HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>>    HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>>    HYPRE BoomerAMG: Maximum row sums 0.9
>>    HYPRE BoomerAMG: Sweeps down         1
>>    HYPRE BoomerAMG: Sweeps up           1
>>    HYPRE BoomerAMG: Sweeps on coarse    1
>>    HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>>    HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>>    HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>>    HYPRE BoomerAMG: Relax weight  (all)      1
>>    HYPRE BoomerAMG: Outer relax weight (all) 1
>>    HYPRE BoomerAMG: Using CF-relaxation
>>    HYPRE BoomerAMG: Measure type        local
>>    HYPRE BoomerAMG: Coarsen type        Falgout
>>    HYPRE BoomerAMG: Interpolation type  classical
>>  linear system matrix = precond matrix:
>>  Matrix Object:   2048 MPI processes
>>    type: mpiaij
>>    rows=4173281, cols=4173281
>>    total: nonzeros=102576661, allocated nonzeros=102576661
>>    total number of mallocs used during MatSetValues calls =0
>>      not using I-node (on process 0) routines
>> --- system solved with PETSc (in 1.005199e+02 seconds)
>>
>>  0 KSP Residual norm 2.368804472986e-01
>>  1 KSP Residual norm 5.676430019132e-02
>>  2 KSP Residual norm 1.898005876002e-02
>>  3 KSP Residual norm 6.193922902926e-03
>>  4 KSP Residual norm 2.008448794493e-03
>>  5 KSP Residual norm 6.390465670228e-04
>>  6 KSP Residual norm 2.157709394389e-04
>>  7 KSP Residual norm 7.295973819979e-05
>>  8 KSP Residual norm 2.358343271482e-05
>>  9 KSP Residual norm 7.489696222066e-06
>> 10 KSP Residual norm 2.390946857593e-06
>> 11 KSP Residual norm 8.068086385140e-07
>> 12 KSP Residual norm 2.706607789749e-07
>> 13 KSP Residual norm 8.636910863376e-08
>> 14 KSP Residual norm 2.761981175852e-08
>> 15 KSP Residual norm 8.755459874369e-09
>> 16 KSP Residual norm 2.708848598341e-09
>> 17 KSP Residual norm 8.968748876265e-10
>> KSP Object: 2048 MPI processes
>>  type: gmres
>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>    GMRES: happy breakdown tolerance 1e-30
>>  maximum iterations=200, initial guess is zero
>>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 2048 MPI processes
>>  type: ml
>>    MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>      Cycles per PCApply=1
>>      Using Galerkin computed coarse grid matrices
>>  Coarse grid solver -- level -------------------------------
>>    KSP Object:    (mg_coarse_)     2048 MPI processes
>>      type: preonly
>>      maximum iterations=1, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object:    (mg_coarse_)     2048 MPI processes
>>      type: redundant
>>        Redundant preconditioner: First (color=0) of 2048 PCs follows
>>      KSP Object:      (mg_coarse_redundant_)       1 MPI processes
>>        type: preonly
>>        maximum iterations=10000, initial guess is zero
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>        left preconditioning
>>        using NONE norm type for convergence test
>>      PC Object:      (mg_coarse_redundant_)       1 MPI processes
>>        type: lu
>>          LU: out-of-place factorization
>>          tolerance for zero pivot 2.22045e-14
>>          using diagonal shift on blocks to prevent zero pivot
>>          matrix ordering: nd
>>          factor fill ratio given 5, needed 4.38504
>>            Factored matrix follows:
>>              Matrix Object:               1 MPI processes
>>                type: seqaij
>>                rows=2055, cols=2055
>>                package used to perform factorization: petsc
>>                total: nonzeros=2476747, allocated nonzeros=2476747
>>                total number of mallocs used during MatSetValues calls =0
>>                  using I-node routines: found 1638 nodes, limit used is
>> 5
>>        linear system matrix = precond matrix:
>>        Matrix Object:         1 MPI processes
>>          type: seqaij
>>          rows=2055, cols=2055
>>          total: nonzeros=564817, allocated nonzeros=1093260
>>          total number of mallocs used during MatSetValues calls =0
>>            not using I-node routines
>>      linear system matrix = precond matrix:
>>      Matrix Object:       2048 MPI processes
>>        type: mpiaij
>>        rows=2055, cols=2055
>>        total: nonzeros=564817, allocated nonzeros=564817
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node (on process 0) routines
>>  Down solver (pre-smoother) on level 1 -------------------------------
>>    KSP Object:    (mg_levels_1_)     2048 MPI processes
>>      type: richardson
>>        Richardson: damping factor=1
>>      maximum iterations=2
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>      left preconditioning
>>      using nonzero initial guess
>>      using NONE norm type for convergence test
>>    PC Object:    (mg_levels_1_)     2048 MPI processes
>>      type: sor
>>        SOR: type = local_symmetric, iterations = 1, local iterations =
>> 1,
>> omega = 1
>>      linear system matrix = precond matrix:
>>      Matrix Object:       2048 MPI processes
>>        type: mpiaij
>>        rows=30194, cols=30194
>>        total: nonzeros=3368414, allocated nonzeros=3368414
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node (on process 0) routines
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  Down solver (pre-smoother) on level 2 -------------------------------
>>    KSP Object:    (mg_levels_2_)     2048 MPI processes
>>      type: richardson
>>        Richardson: damping factor=1
>>      maximum iterations=2
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>      left preconditioning
>>      using nonzero initial guess
>>      using NONE norm type for convergence test
>>    PC Object:    (mg_levels_2_)     2048 MPI processes
>>      type: sor
>>        SOR: type = local_symmetric, iterations = 1, local iterations =
>> 1,
>> omega = 1
>>      linear system matrix = precond matrix:
>>      Matrix Object:       2048 MPI processes
>>        type: mpiaij
>>        rows=531441, cols=531441
>>        total: nonzeros=12476324, allocated nonzeros=12476324
>>        total number of mallocs used during MatSetValues calls =0
>>          not using I-node (on process 0) routines
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  linear system matrix = precond matrix:
>>  Matrix Object:   2048 MPI processes
>>    type: mpiaij
>>    rows=531441, cols=531441
>>    total: nonzeros=12476324, allocated nonzeros=12476324
>>    total number of mallocs used during MatSetValues calls =0
>>      not using I-node (on process 0) routines
>> --- system solved with PETSc (in 2.407844e+02 seconds)
>>
>>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-GAMG
Type: application/octet-stream
Size: 3094403 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/30eed957/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-ML
Type: application/octet-stream
Size: 16021 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/30eed957/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-BoomerAMG
Type: application/octet-stream
Size: 9402 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/30eed957/attachment-0005.obj>

From mfadams at lbl.gov  Tue Oct  8 18:50:13 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Tue, 8 Oct 2013 19:50:13 -0400
Subject: [petsc-users] Problem with AMG packages
In-Reply-To: <50380.82.224.94.31.1381268789.squirrel@webmail.ann.jussieu.fr>
References: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>
	<574A2DCB-6F7B-4279-A4AE-A2A520EE7952@mcs.anl.gov>
	<50380.82.224.94.31.1381268789.squirrel@webmail.ann.jussieu.fr>
Message-ID: <98AF5879-104D-4262-B6F6-EB8167BDCDE6@lbl.gov>

Something is going terrible wrong with the setup in hypre and ML.  hypre's default parameters are not setup well for 3D.  I use:

-pc_hypre_boomeramg_no_CF
-pc_hypre_boomeramg_agg_nl 1
-pc_hypre_boomeramg_coarsen_type HMIS
-pc_hypre_boomeramg_interp_type ext+i

I'm not sure what is going wrong with ML's setup.

GAMG is converging terribly.  Is this just a simple 7-point Laplacian?  It looks like you the eigen estimate is low on the finest grid, which messes up the smoother.  Try running with these parameters and send the output:

-pc_gamg_agg_nsmooths 1 
-pc_gamg_verbose 2
-mg_levels_ksp_type richardson
-mg_levels_pc_type sor

Mark

On Oct 8, 2013, at 5:46 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:

> Please find the log for BoomerAMG, ML and GAMG attached. The set up for
> GAMG doesn't look so bad compared to the other packages, so I'm wondering
> what is going on with those ?
> 
>> 
>>  We need the output from running with -log_summary -pc_mg_log
>> 
>>   Also you can run with PETSc's AMG called GAMG (run with -pc_type gamg)
>> This will give the most useful information about where it is spending
>> the time.
>> 
>> 
>>   Barry
>> 
>> 
>> On Oct 8, 2013, at 4:11 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:
>> 
>>> Dear all,
>>> I'm trying to compare linear solvers for a simple Poisson equation in
>>> 3D.
>>> I thought that MG was the way to go, but looking at my log, the
>>> performance looks abysmal (I know that the matrices are way too small
>>> but
>>> if I go bigger, it just never performs a single iteration ..). Even
>>> though
>>> this is neither the BoomerAMG nor the ML mailing list, could you please
>>> tell me if PETSc sets some default flags that make the setup for those
>>> solvers so slow for this simple problem ? The performance of (G)ASM is
>>> in
>>> comparison much better.
>>> 
>>> Thanks in advance for your help.
>>> 
>>> PS: first the BoomerAMG log, then ML (much more verbose, sorry).
>>> 
>>> 0 KSP Residual norm 1.599647112604e+00
>>> 1 KSP Residual norm 5.450838232404e-02
>>> 2 KSP Residual norm 3.549673478318e-03
>>> 3 KSP Residual norm 2.901826808841e-04
>>> 4 KSP Residual norm 2.574235778729e-05
>>> 5 KSP Residual norm 2.253410171682e-06
>>> 6 KSP Residual norm 1.871067784877e-07
>>> 7 KSP Residual norm 1.681162800670e-08
>>> 8 KSP Residual norm 2.120841512414e-09
>>> KSP Object: 2048 MPI processes
>>> type: gmres
>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>   GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=200, initial guess is zero
>>> tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 2048 MPI processes
>>> type: hypre
>>>   HYPRE BoomerAMG preconditioning
>>>   HYPRE BoomerAMG: Cycle type V
>>>   HYPRE BoomerAMG: Maximum number of levels 25
>>>   HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>>>   HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>>>   HYPRE BoomerAMG: Threshold for strong coupling 0.25
>>>   HYPRE BoomerAMG: Interpolation truncation factor 0
>>>   HYPRE BoomerAMG: Interpolation: max elements per row 0
>>>   HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>>>   HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>>>   HYPRE BoomerAMG: Maximum row sums 0.9
>>>   HYPRE BoomerAMG: Sweeps down         1
>>>   HYPRE BoomerAMG: Sweeps up           1
>>>   HYPRE BoomerAMG: Sweeps on coarse    1
>>>   HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>>>   HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>>>   HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>>>   HYPRE BoomerAMG: Relax weight  (all)      1
>>>   HYPRE BoomerAMG: Outer relax weight (all) 1
>>>   HYPRE BoomerAMG: Using CF-relaxation
>>>   HYPRE BoomerAMG: Measure type        local
>>>   HYPRE BoomerAMG: Coarsen type        Falgout
>>>   HYPRE BoomerAMG: Interpolation type  classical
>>> linear system matrix = precond matrix:
>>> Matrix Object:   2048 MPI processes
>>>   type: mpiaij
>>>   rows=4173281, cols=4173281
>>>   total: nonzeros=102576661, allocated nonzeros=102576661
>>>   total number of mallocs used during MatSetValues calls =0
>>>     not using I-node (on process 0) routines
>>> --- system solved with PETSc (in 1.005199e+02 seconds)
>>> 
>>> 0 KSP Residual norm 2.368804472986e-01
>>> 1 KSP Residual norm 5.676430019132e-02
>>> 2 KSP Residual norm 1.898005876002e-02
>>> 3 KSP Residual norm 6.193922902926e-03
>>> 4 KSP Residual norm 2.008448794493e-03
>>> 5 KSP Residual norm 6.390465670228e-04
>>> 6 KSP Residual norm 2.157709394389e-04
>>> 7 KSP Residual norm 7.295973819979e-05
>>> 8 KSP Residual norm 2.358343271482e-05
>>> 9 KSP Residual norm 7.489696222066e-06
>>> 10 KSP Residual norm 2.390946857593e-06
>>> 11 KSP Residual norm 8.068086385140e-07
>>> 12 KSP Residual norm 2.706607789749e-07
>>> 13 KSP Residual norm 8.636910863376e-08
>>> 14 KSP Residual norm 2.761981175852e-08
>>> 15 KSP Residual norm 8.755459874369e-09
>>> 16 KSP Residual norm 2.708848598341e-09
>>> 17 KSP Residual norm 8.968748876265e-10
>>> KSP Object: 2048 MPI processes
>>> type: gmres
>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>   GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=200, initial guess is zero
>>> tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 2048 MPI processes
>>> type: ml
>>>   MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>     Cycles per PCApply=1
>>>     Using Galerkin computed coarse grid matrices
>>> Coarse grid solver -- level -------------------------------
>>>   KSP Object:    (mg_coarse_)     2048 MPI processes
>>>     type: preonly
>>>     maximum iterations=1, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_coarse_)     2048 MPI processes
>>>     type: redundant
>>>       Redundant preconditioner: First (color=0) of 2048 PCs follows
>>>     KSP Object:      (mg_coarse_redundant_)       1 MPI processes
>>>       type: preonly
>>>       maximum iterations=10000, initial guess is zero
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object:      (mg_coarse_redundant_)       1 MPI processes
>>>       type: lu
>>>         LU: out-of-place factorization
>>>         tolerance for zero pivot 2.22045e-14
>>>         using diagonal shift on blocks to prevent zero pivot
>>>         matrix ordering: nd
>>>         factor fill ratio given 5, needed 4.38504
>>>           Factored matrix follows:
>>>             Matrix Object:               1 MPI processes
>>>               type: seqaij
>>>               rows=2055, cols=2055
>>>               package used to perform factorization: petsc
>>>               total: nonzeros=2476747, allocated nonzeros=2476747
>>>               total number of mallocs used during MatSetValues calls =0
>>>                 using I-node routines: found 1638 nodes, limit used is
>>> 5
>>>       linear system matrix = precond matrix:
>>>       Matrix Object:         1 MPI processes
>>>         type: seqaij
>>>         rows=2055, cols=2055
>>>         total: nonzeros=564817, allocated nonzeros=1093260
>>>         total number of mallocs used during MatSetValues calls =0
>>>           not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=2055, cols=2055
>>>       total: nonzeros=564817, allocated nonzeros=564817
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Down solver (pre-smoother) on level 1 -------------------------------
>>>   KSP Object:    (mg_levels_1_)     2048 MPI processes
>>>     type: richardson
>>>       Richardson: damping factor=1
>>>     maximum iterations=2
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using nonzero initial guess
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_levels_1_)     2048 MPI processes
>>>     type: sor
>>>       SOR: type = local_symmetric, iterations = 1, local iterations =
>>> 1,
>>> omega = 1
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=30194, cols=30194
>>>       total: nonzeros=3368414, allocated nonzeros=3368414
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>> Down solver (pre-smoother) on level 2 -------------------------------
>>>   KSP Object:    (mg_levels_2_)     2048 MPI processes
>>>     type: richardson
>>>       Richardson: damping factor=1
>>>     maximum iterations=2
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using nonzero initial guess
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_levels_2_)     2048 MPI processes
>>>     type: sor
>>>       SOR: type = local_symmetric, iterations = 1, local iterations =
>>> 1,
>>> omega = 1
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=531441, cols=531441
>>>       total: nonzeros=12476324, allocated nonzeros=12476324
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>> linear system matrix = precond matrix:
>>> Matrix Object:   2048 MPI processes
>>>   type: mpiaij
>>>   rows=531441, cols=531441
>>>   total: nonzeros=12476324, allocated nonzeros=12476324
>>>   total number of mallocs used during MatSetValues calls =0
>>>     not using I-node (on process 0) routines
>>> --- system solved with PETSc (in 2.407844e+02 seconds)
>>> 
>>> 
>> 
>> 
> <log-GAMG><log-ML><log-BoomerAMG>


From hsahasra at purdue.edu  Tue Oct  8 18:55:30 2013
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Tue, 8 Oct 2013 19:55:30 -0400 (EDT)
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <877ge3kskv.fsf@mcs.anl.gov>
Message-ID: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>

Hi Jed,

You are right, it doesn't if you use just one PETSc. In our case, we use 2 PETSc libraries (compiled separately, one each for complex and double). And in that case, when the second PETSc library is initialized, cudaSetDevice throws an error. I can't think of any way in which both the PETSc libraries could be initialized with CUDA on the same process.

Do you have any idea?

I can only think of turning CUDA off on one PETSc as a way to avoid this.

Harshad

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>, "Matthew Knepley" <knepley at gmail.com>
Cc: petsc-users at mcs.anl.gov
Sent: Thursday, September 26, 2013 8:37:36 PM
Subject: Re: [petsc-users] cudaSetDevice

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:

> Basically, PetscInitialize is called 2 times, the second time from
> libmesh. 

It should not be.

$ git grep -C3 PetscInitialize src/base/libmesh.C
src/base/libmesh.C-      // Check whether the calling program has already initialized
src/base/libmesh.C-      // PETSc, and avoid duplicate Initialize/Finalize
src/base/libmesh.C-      PetscBool petsc_already_initialized;
src/base/libmesh.C:      ierr = PetscInitialized(&petsc_already_initialized);
src/base/libmesh.C-             CHKERRABORT(libMesh::COMM_WORLD,ierr);
src/base/libmesh.C-      if (petsc_already_initialized != PETSC_TRUE)
src/base/libmesh.C-        libmesh_initialized_petsc = true;
--
src/base/libmesh.C-# else
src/base/libmesh.C-      if (libmesh_initialized_petsc)
src/base/libmesh.C-        {
src/base/libmesh.C:          ierr = PetscInitialize (&argc, const_cast<char***>(&argv), NULL, NULL);
src/base/libmesh.C-              CHKERRABORT(libMesh::COMM_WORLD,ierr);
src/base/libmesh.C-        }
src/base/libmesh.C-# endif

From jedbrown at mcs.anl.gov  Tue Oct  8 19:04:25 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 19:04:25 -0500
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>
References: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>
Message-ID: <87ob6zjomu.fsf@mcs.anl.gov>

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
> You are right, it doesn't if you use just one PETSc. In our case, we
> use 2 PETSc libraries (compiled separately, one each for complex and
> double). 

Please remind us of this when you write.  We don't know everyone in your
group at Purdue and it would simplify our figuring out what could cause
the weird problems you encounter.

> And in that case, when the second PETSc library is initialized,
> cudaSetDevice throws an error. I can't think of any way in which both
> the PETSc libraries could be initialized with CUDA on the same
> process.

This is a general issue of interoperability between multiple libraries
that want to use CUDA.  How can we find out if someone else has already
called cudaSetDevice?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/600f7fc5/attachment.pgp>

From paulmullowney at gmail.com  Tue Oct  8 19:46:35 2013
From: paulmullowney at gmail.com (Paul Mullowney)
Date: Tue, 8 Oct 2013 18:46:35 -0600
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <87ob6zjomu.fsf@mcs.anl.gov>
References: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>
	<87ob6zjomu.fsf@mcs.anl.gov>
Message-ID: <CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>

Perhaps one can do cudaGetDevice() and cudaDeviceReset() in between the two
calls to PetscInitialize in the application code?
Provided no device data was allocated in between the two calls, this might
eliminate the error.

If you are doing this with multiple host threads (especially where the # of
CPU threads > # of GPUs), special care is required.

-Paul


On Tue, Oct 8, 2013 at 6:04 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
> > You are right, it doesn't if you use just one PETSc. In our case, we
> > use 2 PETSc libraries (compiled separately, one each for complex and
> > double).
>
> Please remind us of this when you write.  We don't know everyone in your
> group at Purdue and it would simplify our figuring out what could cause
> the weird problems you encounter.
>
> > And in that case, when the second PETSc library is initialized,
> > cudaSetDevice throws an error. I can't think of any way in which both
> > the PETSc libraries could be initialized with CUDA on the same
> > process.
>
> This is a general issue of interoperability between multiple libraries
> that want to use CUDA.  How can we find out if someone else has already
> called cudaSetDevice?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/a37040f7/attachment.html>

From hsahasra at purdue.edu  Tue Oct  8 19:53:29 2013
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Tue, 8 Oct 2013 20:53:29 -0400 (EDT)
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>
Message-ID: <1804422213.220502.1381280009708.JavaMail.root@mailhub027.itcs.purdue.edu>

I'll try doing that. We will have to take control of cudaSetDevice in our code, instead of PETSc.

Thanks,
Harshad

----- Original Message -----
From: "Paul Mullowney" <paulmullowney at gmail.com>
To: "Jed Brown" <jedbrown at mcs.anl.gov>
Cc: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>, petsc-users at mcs.anl.gov
Sent: Tuesday, October 8, 2013 8:46:35 PM
Subject: Re: [petsc-users] cudaSetDevice



Perhaps one can do cudaGetDevice() and cudaDeviceReset() in between the two calls to PetscInitialize in the application code? 
Provided no device data was allocated in between the two calls, this might eliminate the error. 

If you are doing this with multiple host threads (especially where the # of CPU threads > # of GPUs), special care is required. 

-Paul 



On Tue, Oct 8, 2013 at 6:04 PM, Jed Brown < jedbrown at mcs.anl.gov > wrote: 



Harshad Sahasrabudhe < hsahasra at purdue.edu > writes: 
> You are right, it doesn't if you use just one PETSc. In our case, we 
> use 2 PETSc libraries (compiled separately, one each for complex and 
> double). 

Please remind us of this when you write. We don't know everyone in your 
group at Purdue and it would simplify our figuring out what could cause 
the weird problems you encounter. 


> And in that case, when the second PETSc library is initialized, 
> cudaSetDevice throws an error. I can't think of any way in which both 
> the PETSc libraries could be initialized with CUDA on the same 
> process. 

This is a general issue of interoperability between multiple libraries 
that want to use CUDA. How can we find out if someone else has already 
called cudaSetDevice? 


From jedbrown at mcs.anl.gov  Tue Oct  8 19:54:02 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 08 Oct 2013 19:54:02 -0500
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>
References: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>
	<87ob6zjomu.fsf@mcs.anl.gov>
	<CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>
Message-ID: <87y563i7rp.fsf@mcs.anl.gov>

Paul Mullowney <paulmullowney at gmail.com> writes:

> Perhaps one can do cudaGetDevice() and cudaDeviceReset() in between the two
> calls to PetscInitialize in the application code?
> Provided no device data was allocated in between the two calls, this might
> eliminate the error.

Sure, but how will we actually share the device between libraries?  What
if the other library was not PETSc, but something else, and they also
called cudaSetDevice, but with a different default mapping strategy?

We need an interface that handles this case.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131008/09abe20e/attachment.pgp>

From mfadams at lbl.gov  Tue Oct  8 22:37:47 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Tue, 8 Oct 2013 23:37:47 -0400
Subject: [petsc-users] Problem with AMG packages
In-Reply-To: <50380.82.224.94.31.1381268789.squirrel@webmail.ann.jussieu.fr>
References: <50321.82.224.94.31.1381266691.squirrel@webmail.ann.jussieu.fr>
	<574A2DCB-6F7B-4279-A4AE-A2A520EE7952@mcs.anl.gov>
	<50380.82.224.94.31.1381268789.squirrel@webmail.ann.jussieu.fr>
Message-ID: <F825A06D-E478-4B4A-813D-2930981E95D8@lbl.gov>


On Oct 8, 2013, at 5:46 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:

> Please find the log for BoomerAMG, ML and GAMG attached. The set up for
> GAMG doesn't look so bad compared to the other packages, so I'm wondering
> what is going on with those ?

They all have serious problems.  It just happens that GAMG's problems are not as bad as the others.

> 
>> 
>>  We need the output from running with -log_summary -pc_mg_log
>> 
>>   Also you can run with PETSc's AMG called GAMG (run with -pc_type gamg)
>> This will give the most useful information about where it is spending
>> the time.
>> 
>> 
>>   Barry
>> 
>> 
>> On Oct 8, 2013, at 4:11 PM, Pierre Jolivet <jolivet at ann.jussieu.fr> wrote:
>> 
>>> Dear all,
>>> I'm trying to compare linear solvers for a simple Poisson equation in
>>> 3D.
>>> I thought that MG was the way to go, but looking at my log, the
>>> performance looks abysmal (I know that the matrices are way too small
>>> but
>>> if I go bigger, it just never performs a single iteration ..). Even
>>> though
>>> this is neither the BoomerAMG nor the ML mailing list, could you please
>>> tell me if PETSc sets some default flags that make the setup for those
>>> solvers so slow for this simple problem ? The performance of (G)ASM is
>>> in
>>> comparison much better.
>>> 
>>> Thanks in advance for your help.
>>> 
>>> PS: first the BoomerAMG log, then ML (much more verbose, sorry).
>>> 
>>> 0 KSP Residual norm 1.599647112604e+00
>>> 1 KSP Residual norm 5.450838232404e-02
>>> 2 KSP Residual norm 3.549673478318e-03
>>> 3 KSP Residual norm 2.901826808841e-04
>>> 4 KSP Residual norm 2.574235778729e-05
>>> 5 KSP Residual norm 2.253410171682e-06
>>> 6 KSP Residual norm 1.871067784877e-07
>>> 7 KSP Residual norm 1.681162800670e-08
>>> 8 KSP Residual norm 2.120841512414e-09
>>> KSP Object: 2048 MPI processes
>>> type: gmres
>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>   GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=200, initial guess is zero
>>> tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 2048 MPI processes
>>> type: hypre
>>>   HYPRE BoomerAMG preconditioning
>>>   HYPRE BoomerAMG: Cycle type V
>>>   HYPRE BoomerAMG: Maximum number of levels 25
>>>   HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>>>   HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>>>   HYPRE BoomerAMG: Threshold for strong coupling 0.25
>>>   HYPRE BoomerAMG: Interpolation truncation factor 0
>>>   HYPRE BoomerAMG: Interpolation: max elements per row 0
>>>   HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>>>   HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>>>   HYPRE BoomerAMG: Maximum row sums 0.9
>>>   HYPRE BoomerAMG: Sweeps down         1
>>>   HYPRE BoomerAMG: Sweeps up           1
>>>   HYPRE BoomerAMG: Sweeps on coarse    1
>>>   HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>>>   HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>>>   HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>>>   HYPRE BoomerAMG: Relax weight  (all)      1
>>>   HYPRE BoomerAMG: Outer relax weight (all) 1
>>>   HYPRE BoomerAMG: Using CF-relaxation
>>>   HYPRE BoomerAMG: Measure type        local
>>>   HYPRE BoomerAMG: Coarsen type        Falgout
>>>   HYPRE BoomerAMG: Interpolation type  classical
>>> linear system matrix = precond matrix:
>>> Matrix Object:   2048 MPI processes
>>>   type: mpiaij
>>>   rows=4173281, cols=4173281
>>>   total: nonzeros=102576661, allocated nonzeros=102576661
>>>   total number of mallocs used during MatSetValues calls =0
>>>     not using I-node (on process 0) routines
>>> --- system solved with PETSc (in 1.005199e+02 seconds)
>>> 
>>> 0 KSP Residual norm 2.368804472986e-01
>>> 1 KSP Residual norm 5.676430019132e-02
>>> 2 KSP Residual norm 1.898005876002e-02
>>> 3 KSP Residual norm 6.193922902926e-03
>>> 4 KSP Residual norm 2.008448794493e-03
>>> 5 KSP Residual norm 6.390465670228e-04
>>> 6 KSP Residual norm 2.157709394389e-04
>>> 7 KSP Residual norm 7.295973819979e-05
>>> 8 KSP Residual norm 2.358343271482e-05
>>> 9 KSP Residual norm 7.489696222066e-06
>>> 10 KSP Residual norm 2.390946857593e-06
>>> 11 KSP Residual norm 8.068086385140e-07
>>> 12 KSP Residual norm 2.706607789749e-07
>>> 13 KSP Residual norm 8.636910863376e-08
>>> 14 KSP Residual norm 2.761981175852e-08
>>> 15 KSP Residual norm 8.755459874369e-09
>>> 16 KSP Residual norm 2.708848598341e-09
>>> 17 KSP Residual norm 8.968748876265e-10
>>> KSP Object: 2048 MPI processes
>>> type: gmres
>>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>   GMRES: happy breakdown tolerance 1e-30
>>> maximum iterations=200, initial guess is zero
>>> tolerances:  relative=1e-08, absolute=1e-50, divergence=10000
>>> left preconditioning
>>> using PRECONDITIONED norm type for convergence test
>>> PC Object: 2048 MPI processes
>>> type: ml
>>>   MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>     Cycles per PCApply=1
>>>     Using Galerkin computed coarse grid matrices
>>> Coarse grid solver -- level -------------------------------
>>>   KSP Object:    (mg_coarse_)     2048 MPI processes
>>>     type: preonly
>>>     maximum iterations=1, initial guess is zero
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_coarse_)     2048 MPI processes
>>>     type: redundant
>>>       Redundant preconditioner: First (color=0) of 2048 PCs follows
>>>     KSP Object:      (mg_coarse_redundant_)       1 MPI processes
>>>       type: preonly
>>>       maximum iterations=10000, initial guess is zero
>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>       left preconditioning
>>>       using NONE norm type for convergence test
>>>     PC Object:      (mg_coarse_redundant_)       1 MPI processes
>>>       type: lu
>>>         LU: out-of-place factorization
>>>         tolerance for zero pivot 2.22045e-14
>>>         using diagonal shift on blocks to prevent zero pivot
>>>         matrix ordering: nd
>>>         factor fill ratio given 5, needed 4.38504
>>>           Factored matrix follows:
>>>             Matrix Object:               1 MPI processes
>>>               type: seqaij
>>>               rows=2055, cols=2055
>>>               package used to perform factorization: petsc
>>>               total: nonzeros=2476747, allocated nonzeros=2476747
>>>               total number of mallocs used during MatSetValues calls =0
>>>                 using I-node routines: found 1638 nodes, limit used is
>>> 5
>>>       linear system matrix = precond matrix:
>>>       Matrix Object:         1 MPI processes
>>>         type: seqaij
>>>         rows=2055, cols=2055
>>>         total: nonzeros=564817, allocated nonzeros=1093260
>>>         total number of mallocs used during MatSetValues calls =0
>>>           not using I-node routines
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=2055, cols=2055
>>>       total: nonzeros=564817, allocated nonzeros=564817
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Down solver (pre-smoother) on level 1 -------------------------------
>>>   KSP Object:    (mg_levels_1_)     2048 MPI processes
>>>     type: richardson
>>>       Richardson: damping factor=1
>>>     maximum iterations=2
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using nonzero initial guess
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_levels_1_)     2048 MPI processes
>>>     type: sor
>>>       SOR: type = local_symmetric, iterations = 1, local iterations =
>>> 1,
>>> omega = 1
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=30194, cols=30194
>>>       total: nonzeros=3368414, allocated nonzeros=3368414
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>> Down solver (pre-smoother) on level 2 -------------------------------
>>>   KSP Object:    (mg_levels_2_)     2048 MPI processes
>>>     type: richardson
>>>       Richardson: damping factor=1
>>>     maximum iterations=2
>>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>     left preconditioning
>>>     using nonzero initial guess
>>>     using NONE norm type for convergence test
>>>   PC Object:    (mg_levels_2_)     2048 MPI processes
>>>     type: sor
>>>       SOR: type = local_symmetric, iterations = 1, local iterations =
>>> 1,
>>> omega = 1
>>>     linear system matrix = precond matrix:
>>>     Matrix Object:       2048 MPI processes
>>>       type: mpiaij
>>>       rows=531441, cols=531441
>>>       total: nonzeros=12476324, allocated nonzeros=12476324
>>>       total number of mallocs used during MatSetValues calls =0
>>>         not using I-node (on process 0) routines
>>> Up solver (post-smoother) same as down solver (pre-smoother)
>>> linear system matrix = precond matrix:
>>> Matrix Object:   2048 MPI processes
>>>   type: mpiaij
>>>   rows=531441, cols=531441
>>>   total: nonzeros=12476324, allocated nonzeros=12476324
>>>   total number of mallocs used during MatSetValues calls =0
>>>     not using I-node (on process 0) routines
>>> --- system solved with PETSc (in 2.407844e+02 seconds)
>>> 
>>> 
>> 
>> 
> <log-GAMG><log-ML><log-BoomerAMG>


From christophe.ortiz at ciemat.es  Wed Oct  9 02:33:12 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 9 Oct 2013 09:33:12 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87vc17olr5.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<87ob70rah5.fsf@mcs.anl.gov>
	<CANBrw+6BY7Qsf4QrgKZvmrA-qfEjBkiMjbF1RVT+eDoYWLx3dA@mail.gmail.com>
	<87eh7wospv.fsf@mcs.anl.gov>
	<CANBrw+7Nu10EC0KB7tzV32v_nzySqZ_5vDKTBoxBUVVh65p7Sg@mail.gmail.com>
	<87vc17olr5.fsf@mcs.anl.gov>
Message-ID: <CANBrw+66UDXgr5xnOHtfTTr1ET8E=bgNNppSMvUwbzEvVGQ0bA@mail.gmail.com>

On Tue, Oct 8, 2013 at 4:55 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > What are the names of the subroutines to access these options directly
> > within the code ? I see there is TSAdaptSetStepLimits but this is to set
> > the min and max timestep size.
>
> Sorry, they are not exposed at this time.  They can (and should) be
> added, but you can also register your own controller.  If you'd like to
> help out, you can prepare a patch that adds TSAdaptBasicSetClip() and
> similar functions to src/ts/adapt/impls/basic/adaptbasic.c.
>

Ok. Once I will have implemented my code I can try. I don't promise
anything, but I can try ! Could be interesting.

Saludos,
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/32764e9f/attachment.html>

From rupp at mcs.anl.gov  Wed Oct  9 06:16:17 2013
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Wed, 09 Oct 2013 13:16:17 +0200
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <87y563i7rp.fsf@mcs.anl.gov>
References: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>	<87ob6zjomu.fsf@mcs.anl.gov>	<CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>
	<87y563i7rp.fsf@mcs.anl.gov>
Message-ID: <52553B01.2000809@mcs.anl.gov>

Hi,

 >> Perhaps one can do cudaGetDevice() and cudaDeviceReset() in between 
the two
>> calls to PetscInitialize in the application code?
>> Provided no device data was allocated in between the two calls, this might
>> eliminate the error.
>
> Sure, but how will we actually share the device between libraries?  What
> if the other library was not PETSc, but something else, and they also
> called cudaSetDevice, but with a different default mapping strategy?
>
> We need an interface that handles this case.

Yes, I encountered similar user requests in ViennaCL.

What is a bit tricky is the question of *when* to call cudaSetDevice() 
then. We require users to call PetscInitialize() before anything else, 
so cudaSetDevice() needs to be called from some other place. I like the 
lazy instantiation model, where the GPU backends (OpenCL, CUDA) are 
initialized only when the first object (e.g. a Vec) is created. This 
should provide enough room for all customizations of the CUDA 
initizalization between PetscInitialize() and VecCreate(). I think this 
can be implemented fairly quickly - I'll do it unless there are any 
objections later today.

Best regards,
Karli


From jedbrown at mcs.anl.gov  Wed Oct  9 06:48:08 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 09 Oct 2013 06:48:08 -0500
Subject: [petsc-users] cudaSetDevice
In-Reply-To: <52553B01.2000809@mcs.anl.gov>
References: <1970930671.220357.1381276530074.JavaMail.root@mailhub027.itcs.purdue.edu>
	<87ob6zjomu.fsf@mcs.anl.gov>
	<CAMJ8fwqXOaYYyMH6wx3CaiUZbgAOhQps-TrGVu=O80fyB_OuwA@mail.gmail.com>
	<87y563i7rp.fsf@mcs.anl.gov> <52553B01.2000809@mcs.anl.gov>
Message-ID: <87a9iiis1z.fsf@mcs.anl.gov>

Karl Rupp <rupp at mcs.anl.gov> writes:
> What is a bit tricky is the question of *when* to call cudaSetDevice() 
> then. We require users to call PetscInitialize() before anything else, 
> so cudaSetDevice() needs to be called from some other place. I like the 
> lazy instantiation model, where the GPU backends (OpenCL, CUDA) are 
> initialized only when the first object (e.g. a Vec) is created. 

This should also enable compiling libraries/binaries that support GPUs,
but then running on machines that don't have GPUs or don't have _enough_
for the number of MPI ranks you are using (provided you aren't making
GPU objects).  This is a very desirably property.

When at all possible, defer decisions to run-time when you have more
information about what is needed.

> I think this can be implemented fairly quickly - I'll do it unless
> there are any objections later today.

Thanks!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/217cbfb3/attachment.pgp>

From christophe.ortiz at ciemat.es  Wed Oct  9 07:50:56 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 9 Oct 2013 14:50:56 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
Message-ID: <CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>

Hi,

I am building my Jacobian (IJacobian) with dof > 1 and I am considering to
use either MatSetValuesLocal (what you did in ex10.c from
advection-diffusion) or MatSetValuesBlocked.

Which one do you recommend ?

For the moment I am trying to understand how MatSetValuesBlocked works from
the various examples given in ts tutorials, but there is a mistery for
me...In these examples, the block size is not set. So when m and idxm[] (n
and idxn[]) are passed in, how does PETSc know the correspondance with
global row and column indices in the matrix ?

For instance, ex22.c

for (i=info.xs; i<info.xs+info.xm; i++) {
const PetscReal *k = user->k;
PetscScalar     v[2][2];

v[0][0] = a + k[0]; v[0][1] =  -k[1];
v[1][0] =    -k[0]; v[1][1] = a+k[1];
MatSetValuesBlocked(*Jpre,1,&i,1,&i,&v[0][0],INSERT_VALUES);
}

Here 4 terms due to chemical reactions are passed to the matrix at once.
&i is passed for indice of row and column, which is supposed to be the
indice of the block. But if the blocksize is not set, how PETSc knows to
which row/colum it really corresponds in the matrix ?


For instance, if we take the example from petsc (MatSetValuesBlocked page),
what should I do to pass the block of values 3, 4, 7 and 8 with
MatSetValuesBlocked ?

  1  2  | 3  4
  5  6  | 7  8
    - - - | - - -
 9  10 | 11 12
13 14 | 15 16


Christophe



On Tue, Oct 8, 2013 at 10:57 AM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
> On Mon, Oct 7, 2013 at 11:14 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> On Oct 7, 2013, at 2:49 PM, Christophe Ortiz <christophe.ortiz at ciemat.es>
>> wrote:
>>
>> > Thanks. I got it.
>> >
>> > For the moment I have some general questions.
>> >
>> > - I see that in this example all terms--diffusion and reactions--are
>> put into RHSFunction. There is no IFunction. Is there a reason for that ?
>>  In which cases one should use IFunction or RHSFunction ? In ts/.../ex25.c,
>> diffusion terms are defined in IFunction and reaction terms in RHSFunction.
>> Why not here ?
>>
>>    RHS function is for non-stiff parts of the problem. IFunction for
>> stiff parts.
>>
>>    There reason they are not separated in ex10.c is because I don't know
>> for this problem what is what, so I just put them all in RHSFunction. But
>> use a fully implicit solver.
>>
>
> I understand.
>
>
>> >
>> > - If only RHSFunction is used, how is it possible to define boundary
>> conditions ? For instance, a Dirichlet boundary condition, u(i=0)=3. In
>> ts/.../ex2.c, only RHSFunction is used and the boundary conditions are
>> defined through u_t,
>>
>>    If, for example u is zero on the boundary for all time then you can
>> just set u(t=0) to 0 on the boundary and u_t = 0 on the boundary and the
>> boundary values of u will remain zero.  If you want to do something like u
>> = f(t) on the boundary you can put that into the IFunction.
>>
>
> I see. So I could put either everything in the IFunction, or everything in
> RHS but the two boundaries that cannot be defined in RHS (u=f(t)), in
> IFunction. Correct ?
>
>
>> >
>> > - In this type of problem, which TS scheme is recommended ? TSARKIMEX ?
>>
>>    Beats me.
>>
>> >
>> > - The option TSARKIMEXSetFullyImplicit is used. Why ? Does it help ?
>>
>>     I just wanted to use a fully implicit solver; since I just used
>> RHSFunction by default TSARKIMEX would treat it as explicit which I didn't
>> want.
>>
>
> It makes sense.
>
> Thanks a lot. All this is clear now. I'll probably come back to you soon
> with more specific questions about ordering and matrices.
>
> Christophe
>
>
>>
>>    Barry
>>
>> >
>> > Christophe
>> >
>> > CIEMAT
>> > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
>> > Unidad de Materiales
>> > Edificio 2 - Planta 0 - Despacho 28m
>> > Avenida Complutense 40,
>> > 28040 Madrid, Spain
>> > Tel: +34 91496 2582
>> > Fax: +34 91346 6442
>> >
>> > --
>> > Q
>> > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
>> > Please consider the environment before printing this email.
>> >
>> >
>> > On Mon, Oct 7, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> >   I have put it into the next branch. So just download petsc from the
>> bitbucket site with git and then do git checkout next
>> >
>> >    The code won't work with the current PETSc release.
>> >
>> > On Oct 7, 2013, at 8:16 AM, Christophe Ortiz <
>> christophe.ortiz at ciemat.es> wrote:
>> >
>> > > Hi Barry,
>> > >
>> > > Thanks for the reply.
>> > >
>> > > I had a look at the example ex10.c. But I see it does not fully
>> corresponds to what one can see in the branch. For instance, in the example
>> that is found in the tutorials, there is no RHS defined, no Jacobian, just
>> the IFunction.
>> > >
>> > > Could you please send me the latest version of ex10.c ? Many thanks
>> in advance.
>> > >
>> > > Christophe
>> > >
>> > > CIEMAT
>> > > Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
>> > > Unidad de Materiales
>> > > Edificio 2 - Planta 0 - Despacho 28m
>> > > Avenida Complutense 40,
>> > > 28040 Madrid, Spain
>> > > Tel: +34 91496 2582
>> > > Fax: +34 91346 6442
>> > >
>> > > --
>> > > Q
>> > > Por favor, piense en el medio ambiente antes de imprimir este mensaje.
>> > > Please consider the environment before printing this email.
>> > >
>> > >
>> > > On Mon, Oct 7, 2013 at 2:00 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > >
>> > >   Chris,
>> > >
>> > >     Take a look in the petsc branch barry/wirth-fusion-materials
>> https://bitbucket.org/petsc/petsc/branch/barry%2Fwirth-fusion-materials
>> > > in the file
>> src/ts/examples/tutorials/advection-diffusion-reaction/ex10.c   feel free
>> to ask any questions.
>> > >
>> > >     Barry
>> > >
>> > > On Oct 7, 2013, at 3:03 AM, Christophe Ortiz <
>> christophe.ortiz at ciemat.es> wrote:
>> > >
>> > > > Hi all
>> > > >
>> > > > I need some suggestions to design a code with PETSc.
>> > > >
>> > > > I want to solve a 1D problem composed of several diffusion
>> equations and a lot of ODEs (thousands).
>> > > >
>> > > > - After discretization, the diffusion equations have terms in i-1,
>> i and i+1.
>> > > >
>> > > > - The diffusion equations have additional terms due to couping with
>> all the ODEs. These terms are non-linear.
>> > > >
>> > > > - The terms of the ODEs are local (only depend on node i) but are
>> non-linear.
>> > > >
>> > > > Any advice to design the code ?
>> > > >
>> > > > How should I distribute the terms between IFunction and RHSFunction
>> ?
>> > > >
>> > > > Any special attention to DMDA ? Should I declare just one DMDA with
>> dof >>1 ?
>> > > >
>> > > > Many thanks in advance !
>> > > > Christophe
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/3b480a8a/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed Oct  9 07:56:44 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 09 Oct 2013 07:56:44 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
Message-ID: <87y562hab7.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> Hi,
>
> I am building my Jacobian (IJacobian) with dof > 1 and I am considering to
> use either MatSetValuesLocal (what you did in ex10.c from
> advection-diffusion) or MatSetValuesBlocked.
>
> Which one do you recommend ?

MatSetValuesBlocked or MatSetValuesBlockedLocal.

> For the moment I am trying to understand how MatSetValuesBlocked works from
> the various examples given in ts tutorials, but there is a mistery for
> me...In these examples, the block size is not set. So when m and idxm[] (n
> and idxn[]) are passed in, how does PETSc know the correspondance with
> global row and column indices in the matrix ?
>
> For instance, ex22.c
>
> for (i=info.xs; i<info.xs+info.xm; i++) {
> const PetscReal *k = user->k;
> PetscScalar     v[2][2];
>
> v[0][0] = a + k[0]; v[0][1] =  -k[1];
> v[1][0] =    -k[0]; v[1][1] = a+k[1];
> MatSetValuesBlocked(*Jpre,1,&i,1,&i,&v[0][0],INSERT_VALUES);
> }
>
> Here 4 terms due to chemical reactions are passed to the matrix at once.
> &i is passed for indice of row and column, which is supposed to be the
> indice of the block. But if the blocksize is not set, how PETSc knows to
> which row/colum it really corresponds in the matrix ?

DMCreateMatrix() set the block size to match the "dof" argument to DMDACreate1d.

> For instance, if we take the example from petsc (MatSetValuesBlocked page),
> what should I do to pass the block of values 3, 4, 7 and 8 with
> MatSetValuesBlocked ?
>
>   1  2  | 3  4
>   5  6  | 7  8
>     - - - | - - -
>  9  10 | 11 12
> 13 14 | 15 16

Block row 0, block column 1.  (Count from 0.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/d708b2ae/attachment.pgp>

From christophe.ortiz at ciemat.es  Wed Oct  9 08:13:30 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 9 Oct 2013 15:13:30 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87y562hab7.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
Message-ID: <CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>

On Wed, Oct 9, 2013 at 2:56 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > Hi,
> >
> > I am building my Jacobian (IJacobian) with dof > 1 and I am considering
> to
> > use either MatSetValuesLocal (what you did in ex10.c from
> > advection-diffusion) or MatSetValuesBlocked.
> >
> > Which one do you recommend ?
>
> MatSetValuesBlocked or MatSetValuesBlockedLocal.
>

Ok.


>
> > For the moment I am trying to understand how MatSetValuesBlocked works
> from
> > the various examples given in ts tutorials, but there is a mistery for
> > me...In these examples, the block size is not set. So when m and idxm[]
> (n
> > and idxn[]) are passed in, how does PETSc know the correspondance with
> > global row and column indices in the matrix ?
> >
> > For instance, ex22.c
> >
> > for (i=info.xs; i<info.xs+info.xm; i++) {
> > const PetscReal *k = user->k;
> > PetscScalar     v[2][2];
> >
> > v[0][0] = a + k[0]; v[0][1] =  -k[1];
> > v[1][0] =    -k[0]; v[1][1] = a+k[1];
> > MatSetValuesBlocked(*Jpre,1,&i,1,&i,&v[0][0],INSERT_VALUES);
> > }
> >
> > Here 4 terms due to chemical reactions are passed to the matrix at once.
> > &i is passed for indice of row and column, which is supposed to be the
> > indice of the block. But if the blocksize is not set, how PETSc knows to
> > which row/colum it really corresponds in the matrix ?
>
> DMCreateMatrix() set the block size to match the "dof" argument to
> DMDACreate1d.
>

I suspected it ! It's smart !


>
> > For instance, if we take the example from petsc (MatSetValuesBlocked
> page),
> > what should I do to pass the block of values 3, 4, 7 and 8 with
> > MatSetValuesBlocked ?
> >
> >   1  2  | 3  4
> >   5  6  | 7  8
> >     - - - | - - -
> >  9  10 | 11 12
> > 13 14 | 15 16
>
> Block row 0, block column 1.  (Count from 0.)
>

Thanks !
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/c1c39882/attachment.html>

From christophe.ortiz at ciemat.es  Wed Oct  9 14:04:29 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 9 Oct 2013 21:04:29 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
Message-ID: <CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>

On Wed, Oct 9, 2013 at 3:13 PM, Christophe Ortiz <christophe.ortiz at ciemat.es
> wrote:

>
> On Wed, Oct 9, 2013 at 2:56 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>>
>> > Hi,
>> >
>> > I am building my Jacobian (IJacobian) with dof > 1 and I am considering
>> to
>> > use either MatSetValuesLocal (what you did in ex10.c from
>> > advection-diffusion) or MatSetValuesBlocked.
>> >
>> > Which one do you recommend ?
>>
>> MatSetValuesBlocked or MatSetValuesBlockedLocal.
>>
>
> Ok.
>
>
>>
>> > For the moment I am trying to understand how MatSetValuesBlocked works
>> from
>> > the various examples given in ts tutorials, but there is a mistery for
>> > me...In these examples, the block size is not set. So when m and idxm[]
>> (n
>> > and idxn[]) are passed in, how does PETSc know the correspondance with
>> > global row and column indices in the matrix ?
>> >
>> > For instance, ex22.c
>> >
>> > for (i=info.xs; i<info.xs+info.xm; i++) {
>> > const PetscReal *k = user->k;
>> > PetscScalar     v[2][2];
>> >
>> > v[0][0] = a + k[0]; v[0][1] =  -k[1];
>> > v[1][0] =    -k[0]; v[1][1] = a+k[1];
>> > MatSetValuesBlocked(*Jpre,1,&i,1,&i,&v[0][0],INSERT_VALUES);
>> > }
>> >
>> > Here 4 terms due to chemical reactions are passed to the matrix at once.
>> > &i is passed for indice of row and column, which is supposed to be the
>> > indice of the block. But if the blocksize is not set, how PETSc knows to
>> > which row/colum it really corresponds in the matrix ?
>>
>> DMCreateMatrix() set the block size to match the "dof" argument to
>> DMDACreate1d.
>>
>
> I suspected it ! It's smart !
>
>
>>
>> > For instance, if we take the example from petsc (MatSetValuesBlocked
>> page),
>> > what should I do to pass the block of values 3, 4, 7 and 8 with
>> > MatSetValuesBlocked ?
>> >
>> >   1  2  | 3  4
>> >   5  6  | 7  8
>> >     - - - | - - -
>> >  9  10 | 11 12
>> > 13 14 | 15 16
>>
>> Block row 0, block column 1.  (Count from 0.)
>>
>
>
>
And if with the previous case I would like to pass the values 1, 2, 3, 4,
5, 6, 7 and 8 (the two first rows) what is the matrix val[ ] that I should
pass ? Should it be a matrix val[2][4] ?

If I understand correctly it would be 1 block row, the block row indice
being 0; 2 block columns, the column block indices being 0 and 1:

indices of block rows and block columns:
row = 0
col[0]=0; col[1]=1
MatSetValuesBlocked(*Jpre,1,&row,2,&col,&v[0][0],INSERT_VALUES);

Is it correct ? Is the matrix val correct ?

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/240fed3d/attachment.html>

From mfadams at lbl.gov  Wed Oct  9 15:25:15 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Wed, 9 Oct 2013 16:25:15 -0400
Subject: [petsc-users] SNESSetUp
Message-ID: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>

I see this in SNESSetUp doc:

For basic use of the SNES solvers the user need not explicitly call SNESSetUp(), since these actions will automatically occur during the call to SNESSolve(). However, if one wishes to control this phase separately,
SNESSetUp() should be called after SNESCreate() and optional routines of the form SNESSetXXX(), but before SNESSolve().


I hit this error line in SNESSetUp:

  if (snes->vec_rhs  == snes->vec_sol) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be right hand side vector");

I put this in a debugger and found that these are both 0 (hence equal).  It is not obvious what SNESSetXXX I might be missing.  I've appended what setxxx stuff I'm doing.

Mark

  !  Set function evaluation routine and vector
  call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)

  call SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)

  ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  !  Customize nonlinear solver; set runtime options
  ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type <type>)
  call SNESSetFromOptions(solver%snes,ierr)

  call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/5bbeeaf1/attachment.html>

From knepley at gmail.com  Wed Oct  9 15:33:02 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Oct 2013 15:33:02 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
Message-ID: <CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>

On Wed, Oct 9, 2013 at 3:25 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

> I see this in SNESSetUp doc:
>
> For basic use of the SNES<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNES.html#SNES> solvers
> the user need not explicitly call SNESSetUp<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetUp.html#SNESSetUp>(),
> since these actions will automatically occur during the call to SNESSolve<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSolve.html#SNESSolve>().
> However, if one wishes to control this phase separately,
> SNESSetUp<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetUp.html#SNESSetUp>()
> should be called after SNESCreate<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESCreate.html#SNESCreate>()
> and optional routines of the form SNESSetXXX(), but before SNESSolve<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSolve.html#SNESSolve>
> ().
>
>
> I hit this error line in SNESSetUp:
>
>   if (snes->vec_rhs  == snes->vec_sol)
> SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be right
> hand side vector");
>
> I put this in a debugger and found that these are both 0 (hence equal).
>  It is not obvious what SNESSetXXX I might be missing.  I've appended what
> setxxx stuff I'm doing.
>
> Mark
>
>   !  Set function evaluation routine and vector
>   call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)
>
>   call
> SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)
>
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Customize nonlinear solver; set runtime options
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type
> <type>)
>   call SNESSetFromOptions(solver%snes,ierr)
>
>   call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP
>

Your Fortran is hurting my eyes :)

Is solver%rVec2 == 0?

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/8d43e4c0/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct  9 15:51:40 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 9 Oct 2013 15:51:40 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
Message-ID: <9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>


On Oct 9, 2013, at 2:04 PM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> 
> On Wed, Oct 9, 2013 at 3:13 PM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:
> 
> > For instance, if we take the example from petsc (MatSetValuesBlocked page),
> > what should I do to pass the block of values 3, 4, 7 and 8 with
> > MatSetValuesBlocked ?
> >
> >   1  2  | 3  4
> >   5  6  | 7  8
> >     - - - | - - -
> >  9  10 | 11 12
> > 13 14 | 15 16
> 
> Block row 0, block column 1.  (Count from 0.)
> 
> 
> 
> And if with the previous case I would like to pass the values 1, 2, 3, 4, 5, 6, 7 and 8 (the two first rows) what is the matrix val[ ] that I should pass ? Should it be a matrix val[2][4] ?

   It takes a one dimensional array as input; by default the values should be ordered by row so you would pass in the numerical values 1, 2, 3, 4,5, 6,7,8 in that order. If you wish to have the values ordered by column then you would pass 1 5 2 6 3 7 4 8 after calling MatSetOptions(mat,MAT_ROW_ORIENTED,PETSC_FALSE).

   Note that the ordering you use on the numerical values is the same for both MatSetValues() and MatSetValuesBlocked().

   Barry

> 
> If I understand correctly it would be 1 block row, the block row indice being 0; 2 block columns, the column block indices being 0 and 1:
> 
> indices of block rows and block columns:
> row = 0
> col[0]=0; col[1]=1
> MatSetValuesBlocked(*Jpre,1,&row,2,&col,&v[0][0],INSERT_VALUES);
> 
> Is it correct ? Is the matrix val correct ?
> 
> Christophe
> 


From mfadams at lbl.gov  Wed Oct  9 18:08:05 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Wed, 9 Oct 2013 19:08:05 -0400
Subject: [petsc-users] SNESSetUp
In-Reply-To: <CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
Message-ID: <3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>

> 
> Your Fortran is hurting my eyes :)
> 
> Is solver%rVec2 == 0?

Should be OK.  Here is a fuller listing appended.

SNESSetUp seems wrong in that I don't see how SNESSetUp can work unless a user drills though the interface (or I'm missing some SMESSetXXX method or a side effect).  This code is in SNESSetUp:

  ierr = SNESGetFunction(snes,&snes->vec_func,NULL,NULL);CHKERRQ(ierr);
  if (snes->vec_func == snes->vec_sol) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be function vector");
  if (snes->vec_rhs  == snes->vec_sol) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be right hand side vector");

  if (!snes->vec_sol_update /* && snes->vec_sol */) {
    ierr = VecDuplicate(snes->vec_sol,&snes->vec_sol_update);CHKERRQ(ierr);
    ierr = PetscLogObjectParent((PetscObject)snes,(PetscObject)snes->vec_sol_update);CHKERRQ(ierr);
  }


It seems like first, the (snes->vec_rhs  == snes->vec_sol) test should be moved to SNESSolve where these are apparently set.
Second can we use snes->vec_func instead of snes->vec_sol here?  This would remove all dependance on vec_rhs and vec_sol in SNESSetup.

Mark

  !  Extract global and local vectors from DM; then duplicate for remaining
  !     vectors that are the same types
  call MatGetVecs(solver%KKTmat,solver%xVec2,solver%bVec2,ierr)
  call VecDuplicate(solver%bVec2,solver%rVec2,ierr)

  call SNESSetDM(solver%snes,solver%da,ierr)

  ! call SNESSetApplicationContext(solver%snes,solver,ierr)
  ! call SNESSetApplicationContext(solver%snes,grid,ierr)

  !  Set function evaluation routine and vector
  call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)

  call SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)

  ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  !  Customize nonlinear solver; set runtime options
  ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type <type>)
  call SNESSetFromOptions(solver%snes,ierr)

  call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP



From knepley at gmail.com  Wed Oct  9 19:37:01 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 Oct 2013 19:37:01 -0500
Subject: [petsc-users] ex12 with Neumann BC
In-Reply-To: <524D3D5D.5090301@avignon.inra.fr>
References: <524430F2.2020507@avignon.inra.fr>
	<CAMYG4GmUQYyKLTupFzAd1JsstFykWGQBOq=KE+QAq0EBwUCmYQ@mail.gmail.com>
	<524D3D5D.5090301@avignon.inra.fr>
Message-ID: <CAMYG4Gmph-ye2B_A5QBmP07p4e1zeMd6OPj7Wxw+bxtvTZTZyQ@mail.gmail.com>

On Thu, Oct 3, 2013 at 4:48 AM, Olivier Bonnefon <
olivier.bonnefon at avignon.inra.fr> wrote:

>  Hello,
>
> Thank you for your answer, I'm now able to run the ex12.c with Neumann BC
> (options -bc_type neumann   -interpolate 1).
>
> I have adapted the ex12.c for the 2D-system:
>
> -\Delta u +u =0
>
> It consists in adapting the fem->f0Funcs[0] function and adding the
> jacobian function fem->g0Funcs[0].
> My implementation works for Dirichlet BC.
> With Neumann BC(with options -bc_type neumann   -interpolate 1), the line
> search failed. I think my jacobian functions are corrects, because the
> option "-snes_mf_operator" leads to the same behavior.
>

Sorry this took me a long time. Do you mean that it converges with -snes_mf?


> Do you know what I have missed ?
> In Neumann case, Where is added the 1d-integral along \delta \Omega ?
>

You are correct that I am not doing the correct integral for the Jacobian.
I will put it in soon. That
is why it should work with the FD approximation since the residual is
correct.

  Thanks,

     Matt


> Thanks,
> Olivier Bonnefon
>
>
>
>
>
>
>
> On 09/26/2013 06:54 PM, Matthew Knepley wrote:
>
> On Thu, Sep 26, 2013 at 6:04 AM, Olivier Bonnefon <
> olivier.bonnefon at avignon.inra.fr> wrote:
>
>> Hello,
>>
>> I have implemented my own system from ex12.c. It works with Dirichlet BC,
>> but failed with Neumann one.
>>
>> So, I'm came back to the example /src/snes/example/tutorial/ex12.c, and I
>> tried with Neumann BC:
>>
>> ./ex12 -bc_type NEUMANN
>>
>
>  Here is the full list of tests I run (just checked that it passes in
> 'next'):
>
>
> https://bitbucket.org/petsc/petsc/src/f34a81fe8510aa025c9247a5b14f0fe30e3c0bed/config/builder.py?at=master#cl-175
>
>  Make sure you use an interpolated mesh with Neumann conditions since you
> need faces.
>
>     Matt
>
>
>> This leads to the following crach:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type!
>> [0]PETSC ERROR: Unsupported number of vertices 0 in cell 8 for element
>> geometry computation!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./ex12 on a arch-linux2-c-debug named pcbiom38 by
>> olivierb Thu Sep 26 14:53:32 2013
>> [0]PETSC ERROR: Libraries linked from
>> /home/olivierb/SOFT/petsc-3.4.2/arch-linux2-c-debug/lib
>> [0]PETSC ERROR: Configure run at Thu Sep 26 14:44:42 2013
>> [0]PETSC ERROR: Configure options --with-debugging=1 --download-fiat
>> --download-scientificpython --download-generator --download-triangle
>> --download-ctetgen --download-chaco --download-netcdf --download-hdf5
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: DMPlexComputeCellGeometry() line 732 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexgeometry.c
>> [0]PETSC ERROR: DMPlexComputeResidualFEM() line 558 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexfem.c
>> [0]PETSC ERROR: SNESComputeFunction_DMLocal() line 75 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/snes/utils/dmlocalsnes.c
>> [0]PETSC ERROR: SNESComputeFunction() line 1988 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: SNESSolve() line 3636 in
>> /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>> [0]PETSC ERROR: main() line 582 in
>> "unknowndirectory/"/home/olivierb/solvers/trunk/SandBox/PETSC/LANDSCAPE/REF/ex12.c
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> with errorcode 56.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> --------------------------------------------------------------------------
>>
>> With Gbd, I saw that DMPlexGetConeSize is 0 for the last point.
>>
>> Do I have forget a step to use Neumann BC ?
>>
>> Thanks
>> Olivier Bonnefon
>>
>> --
>> Olivier Bonnefon
>> INRA PACA-Avignon, Unit? BioSP
>> Tel: +33 (0)4 32 72 21 58 <%2B33%20%280%294%2032%2072%2021%2058>
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> Olivier Bonnefon
> INRA PACA-Avignon, Unit? BioSP
> Tel: +33 (0)4 32 72 21 58
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131009/0094f95b/attachment.html>

From christophe.ortiz at ciemat.es  Thu Oct 10 04:50:11 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 10 Oct 2013 11:50:11 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
Message-ID: <CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>

On Wed, Oct 9, 2013 at 10:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Oct 9, 2013, at 2:04 PM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> >
> > On Wed, Oct 9, 2013 at 3:13 PM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
> >
> > > For instance, if we take the example from petsc (MatSetValuesBlocked
> page),
> > > what should I do to pass the block of values 3, 4, 7 and 8 with
> > > MatSetValuesBlocked ?
> > >
> > >   1  2  | 3  4
> > >   5  6  | 7  8
> > >     - - - | - - -
> > >  9  10 | 11 12
> > > 13 14 | 15 16
> >
> > Block row 0, block column 1.  (Count from 0.)
> >
> >
> >
> > And if with the previous case I would like to pass the values 1, 2, 3,
> 4, 5, 6, 7 and 8 (the two first rows) what is the matrix val[ ] that I
> should pass ? Should it be a matrix val[2][4] ?
>
>    It takes a one dimensional array as input; by default the values should
> be ordered by row so you would pass in the numerical values 1, 2, 3, 4,5,
> 6,7,8 in that order. If you wish to have the values ordered by column then
> you would pass 1 5 2 6 3 7 4 8 after calling
> MatSetOptions(mat,MAT_ROW_ORIENTED,PETSC_FALSE).
>
>    Note that the ordering you use on the numerical values is the same for
> both MatSetValues() and MatSetValuesBlocked().
>


I am facing some problems with this when filling in the Jacobian for a
problem of 2 diffusion equations (dof=2). See ex25.c.

If I'm right, then for node i, there should be 3 blocks of 2x2:

X  0  |  Y  0  |  X  0
0  X  |  0  Y  |  0  X

Is it correct ?

First I tried what you said, to pass a one dimensional array:
   row = i;
   cols[0] = i-1; cols[1] = i; cols[2] = i+1;

   j = 0;
   vallin[j] = X; vallin[j++] = 0; vallin[j++] = Y; vallin[j++] = 0;
 vallin[j++] = X; vallin[j++] = 0;

   vallin[j++] = 0; vallin[j++] = X; vallin[j++] = 0; vallin[j++] = Y;
vallin[j++] = 0; vallin[j++] = X;

  ierr = MatSetValuesBlocked(*Jpre,1,&row,3,cols,&vallin[0],INSERT_VALUES);

I hit the following message:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at end of function: Parameter number 3!


Then, I tried a 2-dimensional array. I saw in some examples of ts/tutorials
that it is also possible to pass a multi-dimensional arrays. After all, in
memory the rows are stored contiguously, so I guess it is equivalent to a
one-dimensional array.

      j = 0; k = 0;
      valdiff[j][k] = X; valdiff[j][k++] = 0;
      valdiff[j][k++] = Y; valdiff[j][k++] = 0;
      valdiff[j][k++] = X; valdiff[j][k++] = 0;

      j= 1; k = 0;
      valdiff[j][k] = 0; valdiff[j][k++] = X;
      valdiff[j][k++] = 0; valdiff[j][k++] = Y;
      valdiff[j][k++] = 0; valdiff[j][k++] = X;

  ierr =
MatSetValuesBlocked(*Jpre,1,&row,3,cols,&valdiff[0][0],INSERT_VALUES);

I get the same error message.
What did I do wrong ?

In ex25.c they use a 3-dimensional array:

 const PetscScalar vals[2][3][2] = {{{dxxL,0},{a *hx+dxx0,0},{dxxR,0}},

 {{0,dxxL},{0,a*hx+dxx0},{0,dxxR}}};

and &vals[0][0][0] is passed in MatSetValuesBlocked (....,....,
&vals[0][0][0],...), and it works, though I don't understand how.



Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/1f9a7213/attachment.html>

From christophe.ortiz at ciemat.es  Thu Oct 10 07:27:56 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 10 Oct 2013 14:27:56 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
Message-ID: <CANBrw+5+ggKgRwjGLmwyZXtop5SJmB9vceHbM=zHKXsyvHON4Q@mail.gmail.com>

On Thu, Oct 10, 2013 at 11:50 AM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
> On Wed, Oct 9, 2013 at 10:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> On Oct 9, 2013, at 2:04 PM, Christophe Ortiz <christophe.ortiz at ciemat.es>
>> wrote:
>>
>> >
>> > On Wed, Oct 9, 2013 at 3:13 PM, Christophe Ortiz <
>> christophe.ortiz at ciemat.es> wrote:
>> >
>> > > For instance, if we take the example from petsc (MatSetValuesBlocked
>> page),
>> > > what should I do to pass the block of values 3, 4, 7 and 8 with
>> > > MatSetValuesBlocked ?
>> > >
>> > >   1  2  | 3  4
>> > >   5  6  | 7  8
>> > >     - - - | - - -
>> > >  9  10 | 11 12
>> > > 13 14 | 15 16
>> >
>> > Block row 0, block column 1.  (Count from 0.)
>> >
>> >
>> >
>> > And if with the previous case I would like to pass the values 1, 2, 3,
>> 4, 5, 6, 7 and 8 (the two first rows) what is the matrix val[ ] that I
>> should pass ? Should it be a matrix val[2][4] ?
>>
>>    It takes a one dimensional array as input; by default the values
>> should be ordered by row so you would pass in the numerical values 1, 2, 3,
>> 4,5, 6,7,8 in that order. If you wish to have the values ordered by column
>> then you would pass 1 5 2 6 3 7 4 8 after calling
>> MatSetOptions(mat,MAT_ROW_ORIENTED,PETSC_FALSE).
>>
>>    Note that the ordering you use on the numerical values is the same for
>> both MatSetValues() and MatSetValuesBlocked().
>>
>
>
> I am facing some problems with this when filling in the Jacobian for a
> problem of 2 diffusion equations (dof=2). See ex25.c.
>
> If I'm right, then for node i, there should be 3 blocks of 2x2:
>
> X  0  |  Y  0  |  X  0
> 0  X  |  0  Y  |  0  X
>
> Is it correct ?
>
> First I tried what you said, to pass a one dimensional array:
>    row = i;
>    cols[0] = i-1; cols[1] = i; cols[2] = i+1;
>
>    j = 0;
>    vallin[j] = X; vallin[j++] = 0; vallin[j++] = Y; vallin[j++] = 0;
>  vallin[j++] = X; vallin[j++] = 0;
>
>    vallin[j++] = 0; vallin[j++] = X; vallin[j++] = 0; vallin[j++] = Y;
> vallin[j++] = 0; vallin[j++] = X;
>
>   ierr = MatSetValuesBlocked(*Jpre,1,&row,3,cols,&vallin[0],INSERT_VALUES);
>
> I hit the following message:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
> at end of function: Parameter number 3!
>
>
> Then, I tried a 2-dimensional array. I saw in some examples of
> ts/tutorials that it is also possible to pass a multi-dimensional arrays.
> After all, in memory the rows are stored contiguously, so I guess it is
> equivalent to a one-dimensional array.
>
>       j = 0; k = 0;
>       valdiff[j][k] = X; valdiff[j][k++] = 0;
>       valdiff[j][k++] = Y; valdiff[j][k++] = 0;
>       valdiff[j][k++] = X; valdiff[j][k++] = 0;
>
>       j= 1; k = 0;
>       valdiff[j][k] = 0; valdiff[j][k++] = X;
>       valdiff[j][k++] = 0; valdiff[j][k++] = Y;
>       valdiff[j][k++] = 0; valdiff[j][k++] = X;
>
>   ierr =
> MatSetValuesBlocked(*Jpre,1,&row,3,cols,&valdiff[0][0],INSERT_VALUES);
>
> I get the same error message.
> What did I do wrong ?
>
> In ex25.c they use a 3-dimensional array:
>
>  const PetscScalar vals[2][3][2] = {{{dxxL,0},{a *hx+dxx0,0},{dxxR,0}},
>
>  {{0,dxxL},{0,a*hx+dxx0},{0,dxxR}}};
>
> and &vals[0][0][0] is passed in MatSetValuesBlocked (....,....,
> &vals[0][0][0],...), and it works, though I don't understand how.
>
>
>
>
I found my mistake ! In the indices of the arrays I should have written ++j
and not j++. A good printf of each value helped me to find out...:-).

Now it works.
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/744be8da/attachment-0001.html>

From jedbrown at mcs.anl.gov  Thu Oct 10 07:33:18 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 10 Oct 2013 07:33:18 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
Message-ID: <87hacp2tm9.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> I am facing some problems with this when filling in the Jacobian for a
> problem of 2 diffusion equations (dof=2). See ex25.c.
>
> If I'm right, then for node i, there should be 3 blocks of 2x2:
>
> X  0  |  Y  0  |  X  0
> 0  X  |  0  Y  |  0  X
>
> Is it correct ?
>
> First I tried what you said, to pass a one dimensional array:
>    row = i;
>    cols[0] = i-1; cols[1] = i; cols[2] = i+1;
>
>    j = 0;
>    vallin[j] = X; vallin[j++] = 0; 

The expression j++ in vallin[j++] is post-increment, so the two
statements above both write to vallin[0] and vallin[11] is undefined
(whatever garbage was in the array before).

> Then, I tried a 2-dimensional array. I saw in some examples of ts/tutorials
> that it is also possible to pass a multi-dimensional arrays. After all, in
> memory the rows are stored contiguously, so I guess it is equivalent to a
> one-dimensional array.
>
>       j = 0; k = 0;
>       valdiff[j][k] = X; valdiff[j][k++] = 0;

Same problem here.

>       valdiff[j][k++] = Y; valdiff[j][k++] = 0;
>       valdiff[j][k++] = X; valdiff[j][k++] = 0;
>
>       j= 1; k = 0;
>       valdiff[j][k] = 0; valdiff[j][k++] = X;
>       valdiff[j][k++] = 0; valdiff[j][k++] = Y;
>       valdiff[j][k++] = 0; valdiff[j][k++] = X;
>
>   ierr =
> MatSetValuesBlocked(*Jpre,1,&row,3,cols,&valdiff[0][0],INSERT_VALUES);
>
> I get the same error message.
> What did I do wrong ?
>
> In ex25.c they use a 3-dimensional array:
>
>  const PetscScalar vals[2][3][2] = {{{dxxL,0},{a *hx+dxx0,0},{dxxR,0}},
>
>  {{0,dxxL},{0,a*hx+dxx0},{0,dxxR}}};
>
> and &vals[0][0][0] is passed in MatSetValuesBlocked (....,....,
> &vals[0][0][0],...), and it works, though I don't understand how.

That grouping in the 3D array is the same ordering, but (perhaps) makes
it easier to see which expression refers to which entry of the matrix.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/7e85c833/attachment.pgp>

From christophe.ortiz at ciemat.es  Thu Oct 10 07:47:53 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 10 Oct 2013 14:47:53 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87hacp2tm9.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
Message-ID: <CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>

On Thu, Oct 10, 2013 at 2:33 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > I am facing some problems with this when filling in the Jacobian for a
> > problem of 2 diffusion equations (dof=2). See ex25.c.
> >
> > If I'm right, then for node i, there should be 3 blocks of 2x2:
> >
> > X  0  |  Y  0  |  X  0
> > 0  X  |  0  Y  |  0  X
> >
> > Is it correct ?
> >
> > First I tried what you said, to pass a one dimensional array:
> >    row = i;
> >    cols[0] = i-1; cols[1] = i; cols[2] = i+1;
> >
> >    j = 0;
> >    vallin[j] = X; vallin[j++] = 0;
>
> The expression j++ in vallin[j++] is post-increment, so the two
> statements above both write to vallin[0] and vallin[11] is undefined
> (whatever garbage was in the array before).
>

Thanks.
I just found out few minutes ago :-). Hehe. The point is that I come from
15 years of Fortran...I have to refresh the C that I learnt at Faculty and
that is deeply buried in my memory !



>
> > Then, I tried a 2-dimensional array. I saw in some examples of
> ts/tutorials
> > that it is also possible to pass a multi-dimensional arrays. After all,
> in
> > memory the rows are stored contiguously, so I guess it is equivalent to a
> > one-dimensional array.
> >
> >       j = 0; k = 0;
> >       valdiff[j][k] = X; valdiff[j][k++] = 0;
>
> Same problem here.
>
> >       valdiff[j][k++] = Y; valdiff[j][k++] = 0;
> >       valdiff[j][k++] = X; valdiff[j][k++] = 0;
> >
> >       j= 1; k = 0;
> >       valdiff[j][k] = 0; valdiff[j][k++] = X;
> >       valdiff[j][k++] = 0; valdiff[j][k++] = Y;
> >       valdiff[j][k++] = 0; valdiff[j][k++] = X;
> >
> >   ierr =
> > MatSetValuesBlocked(*Jpre,1,&row,3,cols,&valdiff[0][0],INSERT_VALUES);
> >
> > I get the same error message.
> > What did I do wrong ?
> >
> > In ex25.c they use a 3-dimensional array:
> >
> >  const PetscScalar vals[2][3][2] = {{{dxxL,0},{a *hx+dxx0,0},{dxxR,0}},
> >
> >  {{0,dxxL},{0,a*hx+dxx0},{0,dxxR}}};
> >
> > and &vals[0][0][0] is passed in MatSetValuesBlocked (....,....,
> > &vals[0][0][0],...), and it works, though I don't understand how.
>
> That grouping in the 3D array is the same ordering, but (perhaps) makes
> it easier to see which expression refers to which entry of the matrix.
>

Yes, probably.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/f36be5a4/attachment.html>

From knepley at gmail.com  Thu Oct 10 12:07:29 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Oct 2013 12:07:29 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
Message-ID: <CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>

On Wed, Oct 9, 2013 at 6:08 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

> >
> > Your Fortran is hurting my eyes :)
> >
> > Is solver%rVec2 == 0?
>
> Should be OK.  Here is a fuller listing appended.
>
> SNESSetUp seems wrong in that I don't see how SNESSetUp can work unless a
> user drills though the interface (or I'm missing some SMESSetXXX method or
> a side effect).  This code is in SNESSetUp:
>
>   ierr = SNESGetFunction(snes,&snes->vec_func,NULL,NULL);CHKERRQ(ierr);
>   if (snes->vec_func == snes->vec_sol)
> SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be
> function vector");
>   if (snes->vec_rhs  == snes->vec_sol)
> SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_IDN,"Solution vector cannot be right
> hand side vector");
>
>   if (!snes->vec_sol_update /* && snes->vec_sol */) {
>     ierr = VecDuplicate(snes->vec_sol,&snes->vec_sol_update);CHKERRQ(ierr);
>     ierr =
> PetscLogObjectParent((PetscObject)snes,(PetscObject)snes->vec_sol_update);CHKERRQ(ierr);
>   }
>
>
> It seems like first, the (snes->vec_rhs  == snes->vec_sol) test should be
> moved to SNESSolve where these are apparently set.
>

This is not good enough since the next lines will fail if snes->vec_sol is
NULL. I think we do not to substitute vec_func for vec_sol, but
we also need to move creation of the update vector since it must be the
size of the solution. Will do it now.

   Matt


> Second can we use snes->vec_func instead of snes->vec_sol here?  This
> would remove all dependance on vec_rhs and vec_sol in SNESSetup.
>
> Mark
>
>   !  Extract global and local vectors from DM; then duplicate for remaining
>   !     vectors that are the same types
>   call MatGetVecs(solver%KKTmat,solver%xVec2,solver%bVec2,ierr)
>   call VecDuplicate(solver%bVec2,solver%rVec2,ierr)
>
>   call SNESSetDM(solver%snes,solver%da,ierr)
>
>   ! call SNESSetApplicationContext(solver%snes,solver,ierr)
>   ! call SNESSetApplicationContext(solver%snes,grid,ierr)
>
>   !  Set function evaluation routine and vector
>   call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)
>
>   call
> SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)
>
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Customize nonlinear solver; set runtime options
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type
> <type>)
>   call SNESSetFromOptions(solver%snes,ierr)
>
>   call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/39388046/attachment.html>

From mfadams at lbl.gov  Thu Oct 10 12:28:02 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Thu, 10 Oct 2013 13:28:02 -0400
Subject: [petsc-users] SNESSetUp
In-Reply-To: <CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
Message-ID: <48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>

> 
> It seems like first, the (snes->vec_rhs  == snes->vec_sol) test should be moved to SNESSolve where these are apparently set.
> 
> This is not good enough since the next lines will fail if snes->vec_sol is NULL.

That's what I suggested below.  I can test this now if you like.

Mark

> I think we do not to substitute vec_func for vec_sol, but
> we also need to move creation of the update vector since it must be the size of the solution. Will do it now.
> 
>    Matt
>  
> Second can we use snes->vec_func instead of snes->vec_sol here?  This would remove all dependance on vec_rhs and vec_sol in SNESSetup.
> 
> Mark
> 
>   !  Extract global and local vectors from DM; then duplicate for remaining
>   !     vectors that are the same types
>   call MatGetVecs(solver%KKTmat,solver%xVec2,solver%bVec2,ierr)
>   call VecDuplicate(solver%bVec2,solver%rVec2,ierr)
> 
>   call SNESSetDM(solver%snes,solver%da,ierr)
> 
>   ! call SNESSetApplicationContext(solver%snes,solver,ierr)
>   ! call SNESSetApplicationContext(solver%snes,grid,ierr)
> 
>   !  Set function evaluation routine and vector
>   call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)
> 
>   call SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)
> 
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Customize nonlinear solver; set runtime options
>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>   !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type <type>)
>   call SNESSetFromOptions(solver%snes,ierr)
> 
>   call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/c6fcdd88/attachment.html>

From knepley at gmail.com  Thu Oct 10 12:32:07 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 Oct 2013 12:32:07 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
Message-ID: <CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>

On Thu, Oct 10, 2013 at 12:28 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

>
>> It seems like first, the (snes->vec_rhs  == snes->vec_sol) test should be
>> moved to SNESSolve where these are apparently set.
>>
>
> This is not good enough since the next lines will fail if snes->vec_sol is
> NULL.
>
>
> That's what I suggested below.  I can test this now if you like.
>

Excellent. I just pushed to next.

  Matt


> Mark
>
> I think we do not to substitute vec_func for vec_sol, but
> we also need to move creation of the update vector since it must be the
> size of the solution. Will do it now.
>
>    Matt
>
>
>> Second can we use snes->vec_func instead of snes->vec_sol here?  This
>> would remove all dependance on vec_rhs and vec_sol in SNESSetup.
>>
>> Mark
>>
>>   !  Extract global and local vectors from DM; then duplicate for
>> remaining
>>   !     vectors that are the same types
>>   call MatGetVecs(solver%KKTmat,solver%xVec2,solver%bVec2,ierr)
>>   call VecDuplicate(solver%bVec2,solver%rVec2,ierr)
>>
>>   call SNESSetDM(solver%snes,solver%da,ierr)
>>
>>   ! call SNESSetApplicationContext(solver%snes,solver,ierr)
>>   ! call SNESSetApplicationContext(solver%snes,grid,ierr)
>>
>>   !  Set function evaluation routine and vector
>>   call SNESSetFunction(solver%snes,solver%rVec2,FormFunction,solver,ierr)
>>
>>   call
>> SNESSetJacobian(solver%snes,solver%KKTmat,solver%KKTmat,FormJacobian,solver,ierr)
>>
>>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>   !  Customize nonlinear solver; set runtime options
>>   ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>   !  Set runtime options (e.g., -snes_monitor -snes_rtol <rtol> -ksp_type
>> <type>)
>>   call SNESSetFromOptions(solver%snes,ierr)
>>
>>   call SNESSetUp(solver%snes,ierr) ! pre setup, same as (old) KSP
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/9137d0cc/attachment.html>

From mfadams at lbl.gov  Thu Oct 10 14:37:00 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Thu, 10 Oct 2013 15:37:00 -0400
Subject: [petsc-users] SNESSetUp
In-Reply-To: <CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
	<CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
Message-ID: <760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>

> 
> That's what I suggested below.  I can test this now if you like.
> 
> Excellent. I just pushed to next.

It is almost done but SNESSetWorkVecs tries to use vec_sol to create work vectors.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131010/787e7b13/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 10 14:41:10 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 10 Oct 2013 14:41:10 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
	<CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
	<760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>
Message-ID: <38980DE7-CF48-4782-A6E7-218A25E3EF68@mcs.anl.gov>


On Oct 10, 2013, at 2:37 PM, "Mark F. Adams" <mfadams at lbl.gov> wrote:

>> 
>> That's what I suggested below.  I can test this now if you like.
>> 
>> Excellent. I just pushed to next.
> 
> It is almost done but SNESSetWorkVecs tries to use vec_sol to create work vectors.
> 

   If SNES has a DM then it should be trying to get its work vectors from that.

   Barry


From asmund.ervik at ntnu.no  Fri Oct 11 07:50:50 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Fri, 11 Oct 2013 14:50:50 +0200
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <877gdpt6s0.fsf@mcs.anl.gov>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<87d2nht8o2.fsf@mcs.anl.gov> <5252D5E7.8050906@ntnu.no>
	<877gdpt6s0.fsf@mcs.anl.gov>
Message-ID: <5257F42A.4030808@ntnu.no>

Hi again,

I am using a 3D DMDA and I want to extract the bounds for the local
domain. For some reason, DMDAGetGhostCorners gives me the wrong
numbers. Consider this code:

    call DMCreateLocalVector(SolScal,Lpressure,ierr)
    call DMDAVecGetArrayF90(SolScal,Lpressure,array,ierr)
    tmp=shape(array)
    call DMDAVecRestoreArrayF90(SolScal,Lpressure,array,ierr)
    call DMDAGetGhostCorners(SolScal,i,j,k,m,n,p,ierr)
    write(*,*) tmp(2:)
    write(*,*) m-i,n-j,p-k

I.e. I create a local array and inquire about it's shape, and then I
get info from DMDAGetGhostCorners. So the two last lines should print
the same, but they don't:

mpirun -np 2 meph
          25          26          50
          25           2          50
          25          26          50
          25          26          50

Note the difference between the first and second lines.

Am I using DMDAGetGhostCorners wrong here, or what?

Best regards,
?smund Ervik



On 07. okt. 2013 17:55, Jed Brown wrote:
> ?smund Ervik <asmund.ervik at ntnu.no> writes:
>> Thanks for this comment; I have been suspecting this is a bad
>> idea, nice to get confirmation. I will follow your advice.
> 
> Cool.
> 
>> Other than that, does the mapping from local to global using
>> Fortran assumed-shape arrays make sense, or is there a better way
>> of doing it?
> 
> It looks okay to me.
> 

From knepley at gmail.com  Fri Oct 11 08:02:37 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Oct 2013 08:02:37 -0500
Subject: [petsc-users] Writing a domain decomposition code with PETSc
In-Reply-To: <5257F42A.4030808@ntnu.no>
References: <3ago0hmh8msq28oe6xdqo9p4.1380912676104@email.android.com>
	<CAMYG4GnJtPodB667uf8VncKS3rsxOge_nTDuctM3PZp+UWheuw@mail.gmail.com>
	<ehtmdu6s773wlsqtjxx5xo1s.1380913806224@email.android.com>
	<CAMYG4Gk=ebUqHuWbUU_gbDzAzb=iXorcOHdn0_Ay=fSr6ZVfxQ@mail.gmail.com>
	<87d2nkzs5f.fsf@mcs.anl.gov> <5252A966.4010406@ntnu.no>
	<87d2nht8o2.fsf@mcs.anl.gov> <5252D5E7.8050906@ntnu.no>
	<877gdpt6s0.fsf@mcs.anl.gov> <5257F42A.4030808@ntnu.no>
Message-ID: <CAMYG4GmhMnPegjKfWXH=8R+FadF=oo=c22BrNTi5JFHUmHFCKA@mail.gmail.com>

On Fri, Oct 11, 2013 at 7:50 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

> Hi again,
>
> I am using a 3D DMDA and I want to extract the bounds for the local
> domain. For some reason, DMDAGetGhostCorners gives me the wrong
> numbers. Consider this code:
>
>     call DMCreateLocalVector(SolScal,Lpressure,ierr)
>     call DMDAVecGetArrayF90(SolScal,Lpressure,array,ierr)
>     tmp=shape(array)
>     call DMDAVecRestoreArrayF90(SolScal,Lpressure,array,ierr)
>     call DMDAGetGhostCorners(SolScal,i,j,k,m,n,p,ierr)
>     write(*,*) tmp(2:)
>     write(*,*) m-i,n-j,p-k
>
> I.e. I create a local array and inquire about it's shape, and then I
> get info from DMDAGetGhostCorners. So the two last lines should print
> the same, but they don't:
>
> mpirun -np 2 meph
>           25          26          50
>           25           2          50
>           25          26          50
>           25          26          50
>
> Note the difference between the first and second lines.
>
> Am I using DMDAGetGhostCorners wrong here, or what?
>

Yes. n-j does not mean anything. n is the local size, whereas j is the
global y offset.

   Matt


> Best regards,
> ?smund Ervik
>
>
>
> On 07. okt. 2013 17:55, Jed Brown wrote:
> > ?smund Ervik <asmund.ervik at ntnu.no> writes:
> >> Thanks for this comment; I have been suspecting this is a bad
> >> idea, nice to get confirmation. I will follow your advice.
> >
> > Cool.
> >
> >> Other than that, does the mapping from local to global using
> >> Fortran assumed-shape arrays make sense, or is there a better way
> >> of doing it?
> >
> > It looks okay to me.
> >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/56606123/attachment.html>

From bisheshkh at gmail.com  Fri Oct 11 08:09:00 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 11 Oct 2013 15:09:00 +0200
Subject: [petsc-users] reordering the dmda dof
Message-ID: <CAEhex8hZvokqYreWzXd=xA=XW0AbNLOiXfzGKwXqetGASE1McQ@mail.gmail.com>

Dear all,
By default, dmda puts all dof at a given node logically in consecutive
order. The corresponding vectors and matrices created also follow this
order. What should I do to change it so that it puts all the grid nodes for
a 1st dof consecutively, then do the same for 2nd dof and so on. I also
want the same behavior for the matrix it creates so that the rows
corresponding to a particular dof are put in a consecutive manner instead
of interleaving different dofs.

Thanks,
Bishesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/52e79654/attachment.html>

From knepley at gmail.com  Fri Oct 11 08:13:42 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Oct 2013 08:13:42 -0500
Subject: [petsc-users] reordering the dmda dof
In-Reply-To: <CAEhex8hZvokqYreWzXd=xA=XW0AbNLOiXfzGKwXqetGASE1McQ@mail.gmail.com>
References: <CAEhex8hZvokqYreWzXd=xA=XW0AbNLOiXfzGKwXqetGASE1McQ@mail.gmail.com>
Message-ID: <CAMYG4G=kQ77GP6AM4JuebTNU_kC3RnVTHCJDfcVE=dkt_G7Wvw@mail.gmail.com>

On Fri, Oct 11, 2013 at 8:09 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> Dear all,
> By default, dmda puts all dof at a given node logically in consecutive
> order. The corresponding vectors and matrices created also follow this
> order. What should I do to change it so that it puts all the grid nodes for
> a 1st dof consecutively, then do the same for 2nd dof and so on. I also
> want the same behavior for the matrix it creates so that the rows
> corresponding to a particular dof are put in a consecutive manner instead
> of interleaving different dofs.
>

What do you really want to accomplish? This ordering is disastrous for
cache architectures.

1) If you want segregated solves, use PCFIELDSPLIT.

2) If you want to mess around with a single field, use
VecStrideGather/Scatter():
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html

  Thanks,

     Matt

Thanks,
> Bishesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/3ce3e32b/attachment.html>

From bisheshkh at gmail.com  Fri Oct 11 08:25:55 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 11 Oct 2013 15:25:55 +0200
Subject: [petsc-users] reordering the dmda dof
In-Reply-To: <CAMYG4G=kQ77GP6AM4JuebTNU_kC3RnVTHCJDfcVE=dkt_G7Wvw@mail.gmail.com>
References: <CAEhex8hZvokqYreWzXd=xA=XW0AbNLOiXfzGKwXqetGASE1McQ@mail.gmail.com>
	<CAMYG4G=kQ77GP6AM4JuebTNU_kC3RnVTHCJDfcVE=dkt_G7Wvw@mail.gmail.com>
Message-ID: <CAEhex8ikqZB=P6_Pj6FW=XZDBq3xcBxE6JgiU-gHBaYQnYePvw@mail.gmail.com>

On Fri, Oct 11, 2013 at 3:13 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 11, 2013 at 8:09 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> Dear all,
>> By default, dmda puts all dof at a given node logically in consecutive
>> order. The corresponding vectors and matrices created also follow this
>> order. What should I do to change it so that it puts all the grid nodes for
>> a 1st dof consecutively, then do the same for 2nd dof and so on. I also
>> want the same behavior for the matrix it creates so that the rows
>> corresponding to a particular dof are put in a consecutive manner instead
>> of interleaving different dofs.
>>
>
> What do you really want to accomplish? This ordering is disastrous for
> cache architectures.
>

Thanks Matt, but I was trying to check if there is any problem in a matrix
I created. I had my old prototype code in MATLAB where I created a matrix
by having a block of rows for each dof. So as a quick test, I wanted to
write the matrix I created in Petsc to a file and read it with matlab and
compare the two matrices, taking a difference would help me spot errors
quickly.
Please let me know if there is a quick way to do that. And if there are
better ways to track down possible errors in the matrix creation by having
better visualization of elements of the matrix created in petsc (without
going to matlab), I'd be happy to learn that too.



> 1) If you want segregated solves, use PCFIELDSPLIT.
>
> 2) If you want to mess around with a single field, use
> VecStrideGather/Scatter():
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecStrideGather.html
>
>   Thanks,
>
>      Matt
>
> Thanks,
>> Bishesh
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/91ed6894/attachment.html>

From jedbrown at mcs.anl.gov  Fri Oct 11 08:41:23 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 11 Oct 2013 08:41:23 -0500
Subject: [petsc-users] reordering the dmda dof
In-Reply-To: <CAEhex8ikqZB=P6_Pj6FW=XZDBq3xcBxE6JgiU-gHBaYQnYePvw@mail.gmail.com>
References: <CAEhex8hZvokqYreWzXd=xA=XW0AbNLOiXfzGKwXqetGASE1McQ@mail.gmail.com>
	<CAMYG4G=kQ77GP6AM4JuebTNU_kC3RnVTHCJDfcVE=dkt_G7Wvw@mail.gmail.com>
	<CAEhex8ikqZB=P6_Pj6FW=XZDBq3xcBxE6JgiU-gHBaYQnYePvw@mail.gmail.com>
Message-ID: <874n8nzzzw.fsf@mcs.anl.gov>

Bishesh Khanal <bisheshkh at gmail.com> writes:
> Thanks Matt, but I was trying to check if there is any problem in a matrix
> I created. I had my old prototype code in MATLAB where I created a matrix
> by having a block of rows for each dof. So as a quick test, I wanted to
> write the matrix I created in Petsc to a file and read it with matlab and
> compare the two matrices, taking a difference would help me spot errors
> quickly.

Just permute the matrix, either using ISCreateGeneral and MatPermute or
in Matlab by creating P = [1, m+1, 2*m+1, 2, m+2, 2*m+2, ...] and then
looking at A(P,P).

> Please let me know if there is a quick way to do that. And if there are
> better ways to track down possible errors in the matrix creation by having
> better visualization of elements of the matrix created in petsc (without
> going to matlab), I'd be happy to learn that too.

If this is the Jacobian of something, use -snes_type test to compare
what you assembled to the matrix obtained by finite differencing the
residual function.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/e905d814/attachment-0001.pgp>

From olivier.bonnefon at avignon.inra.fr  Fri Oct 11 09:01:38 2013
From: olivier.bonnefon at avignon.inra.fr (Olivier Bonnefon)
Date: Fri, 11 Oct 2013 16:01:38 +0200
Subject: [petsc-users] ex12 with Neumann BC
In-Reply-To: <CAMYG4Gmph-ye2B_A5QBmP07p4e1zeMd6OPj7Wxw+bxtvTZTZyQ@mail.gmail.com>
References: <524430F2.2020507@avignon.inra.fr>
	<CAMYG4GmUQYyKLTupFzAd1JsstFykWGQBOq=KE+QAq0EBwUCmYQ@mail.gmail.com>
	<524D3D5D.5090301@avignon.inra.fr>
	<CAMYG4Gmph-ye2B_A5QBmP07p4e1zeMd6OPj7Wxw+bxtvTZTZyQ@mail.gmail.com>
Message-ID: <525804C2.3000101@avignon.inra.fr>

Hello,

I now able to simulate a two fields non-linear system with fem, and I 
get a good speed up with Dirichlet BC. Thanks for your help.

About Neumann BC, I need to add "-ksp_type fgmres", else the line search 
failed (even with -snes_mf).
More over, the Neumann version works only in sequential. Indeed, in 
parallel, the boundary conditions are applied inside the domain. I guess 
it is because the label are automatically add after the mesh 
partitioning. This problem doesn't appear in ex12 because there is a 
test in the plugin function, but it is not possible to do the same thing 
with a complex shape. I don't know what it must be done.

In Neumann case, the 1D-integral apears only in the residual part (ie 
the right side hand of the system). Is it done in the 
FEMIntegrateBdResidualBatch function ?

Thanks
Olivier B



On 10/10/2013 02:37 AM, Matthew Knepley wrote:
> On Thu, Oct 3, 2013 at 4:48 AM, Olivier Bonnefon 
> <olivier.bonnefon at avignon.inra.fr 
> <mailto:olivier.bonnefon at avignon.inra.fr>> wrote:
>
>     Hello,
>
>     Thank you for your answer, I'm now able to run the ex12.c with
>     Neumann BC (options -bc_type neumann -interpolate 1).
>
>     I have adapted the ex12.c for the 2D-system:
>
>     -\Delta u +u =0
>
>     It consists in adapting the fem->f0Funcs[0] function and adding
>     the jacobian function fem->g0Funcs[0].
>     My implementation works for Dirichlet BC.
>     With Neumann BC(with options -bc_type neumann -interpolate 1), the
>     line search failed. I think my jacobian functions are corrects,
>     because the option "-snes_mf_operator" leads to the same behavior.
>
>
> Sorry this took me a long time. Do you mean that it converges with 
> -snes_mf?
>
>     Do you know what I have missed ?
>     In Neumann case, Where is added the 1d-integral along \delta \Omega ?
>
>
> You are correct that I am not doing the correct integral for the 
> Jacobian. I will put it in soon. That
> is why it should work with the FD approximation since the residual is 
> correct.
>
>   Thanks,
>
>      Matt
>
>     Thanks,
>     Olivier Bonnefon
>
>
>
>
>
>
>
>     On 09/26/2013 06:54 PM, Matthew Knepley wrote:
>>     On Thu, Sep 26, 2013 at 6:04 AM, Olivier Bonnefon
>>     <olivier.bonnefon at avignon.inra.fr
>>     <mailto:olivier.bonnefon at avignon.inra.fr>> wrote:
>>
>>         Hello,
>>
>>         I have implemented my own system from ex12.c. It works with
>>         Dirichlet BC, but failed with Neumann one.
>>
>>         So, I'm came back to the example
>>         /src/snes/example/tutorial/ex12.c, and I tried with Neumann BC:
>>
>>         ./ex12 -bc_type NEUMANN
>>
>>
>>     Here is the full list of tests I run (just checked that it passes
>>     in 'next'):
>>
>>     https://bitbucket.org/petsc/petsc/src/f34a81fe8510aa025c9247a5b14f0fe30e3c0bed/config/builder.py?at=master#cl-175
>>
>>     Make sure you use an interpolated mesh with Neumann conditions
>>     since you need faces.
>>
>>        Matt
>>
>>         This leads to the following crach:
>>
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ------------------------------------
>>         [0]PETSC ERROR: No support for this operation for this object
>>         type!
>>         [0]PETSC ERROR: Unsupported number of vertices 0 in cell 8
>>         for element geometry computation!
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: ./ex12 on a arch-linux2-c-debug named
>>         pcbiom38 by olivierb Thu Sep 26 14:53:32 2013
>>         [0]PETSC ERROR: Libraries linked from
>>         /home/olivierb/SOFT/petsc-3.4.2/arch-linux2-c-debug/lib
>>         [0]PETSC ERROR: Configure run at Thu Sep 26 14:44:42 2013
>>         [0]PETSC ERROR: Configure options --with-debugging=1
>>         --download-fiat --download-scientificpython
>>         --download-generator --download-triangle --download-ctetgen
>>         --download-chaco --download-netcdf --download-hdf5
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: DMPlexComputeCellGeometry() line 732 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexgeometry.c
>>         [0]PETSC ERROR: DMPlexComputeResidualFEM() line 558 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/dm/impls/plex/plexfem.c
>>         [0]PETSC ERROR: SNESComputeFunction_DMLocal() line 75 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/snes/utils/dmlocalsnes.c
>>         [0]PETSC ERROR: SNESComputeFunction() line 1988 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>>         [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/snes/impls/ls/ls.c
>>         [0]PETSC ERROR: SNESSolve() line 3636 in
>>         /home/olivierb/SOFT/petsc-3.4.2/src/snes/interface/snes.c
>>         [0]PETSC ERROR: main() line 582 in
>>         "unknowndirectory/"/home/olivierb/solvers/trunk/SandBox/PETSC/LANDSCAPE/REF/ex12.c
>>         --------------------------------------------------------------------------
>>         MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>         with errorcode 56.
>>
>>         NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>         processes.
>>         You may or may not see output from other processes, depending on
>>         exactly when Open MPI kills them.
>>         --------------------------------------------------------------------------
>>
>>         With Gbd, I saw that DMPlexGetConeSize is 0 for the last point.
>>
>>         Do I have forget a step to use Neumann BC ?
>>
>>         Thanks
>>         Olivier Bonnefon
>>
>>         -- 
>>         Olivier Bonnefon
>>         INRA PACA-Avignon, Unit? BioSP
>>         Tel: +33 (0)4 32 72 21 58
>>         <tel:%2B33%20%280%294%2032%2072%2021%2058>
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>     -- 
>     Olivier Bonnefon
>     INRA PACA-Avignon, Unit? BioSP
>     Tel:+33 (0)4 32 72 21 58  <tel:%2B33%20%280%294%2032%2072%2021%2058>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
Olivier Bonnefon
INRA PACA-Avignon, Unit? BioSP
Tel: +33 (0)4 32 72 21 58

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/c2fd2e16/attachment.html>

From knepley at gmail.com  Fri Oct 11 13:11:34 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Oct 2013 13:11:34 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <38980DE7-CF48-4782-A6E7-218A25E3EF68@mcs.anl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
	<CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
	<760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>
	<38980DE7-CF48-4782-A6E7-218A25E3EF68@mcs.anl.gov>
Message-ID: <CAMYG4GmgAcpJOoZGhEubApQ6PN1qGLyHM4WKvLXfABV1zUhcLg@mail.gmail.com>

On Thu, Oct 10, 2013 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Oct 10, 2013, at 2:37 PM, "Mark F. Adams" <mfadams at lbl.gov> wrote:
>
> >>
> >> That's what I suggested below.  I can test this now if you like.
> >>
> >> Excellent. I just pushed to next.
> >
> > It is almost done but SNESSetWorkVecs tries to use vec_sol to create
> work vectors.
> >
>
>    If SNES has a DM then it should be trying to get its work vectors from
> that.


Now a SNES always has a DM. Changed.

  Matt


>
>    Barry
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/d0b68ca1/attachment.html>

From mfadams at lbl.gov  Fri Oct 11 14:59:28 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Fri, 11 Oct 2013 15:59:28 -0400
Subject: [petsc-users] SNESSetUp
In-Reply-To: <CAMYG4GmgAcpJOoZGhEubApQ6PN1qGLyHM4WKvLXfABV1zUhcLg@mail.gmail.com>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
	<CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
	<760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>
	<38980DE7-CF48-4782-A6E7-218A25E3EF68@mcs.anl.gov>
	<CAMYG4GmgAcpJOoZGhEubApQ6PN1qGLyHM4WKvLXfABV1zUhcLg@mail.gmail.com>
Message-ID: <B03C44F8-EBC4-4537-8078-771D12511738@lbl.gov>

I am working on fixing this in Newton, getting the DM.  I've tested and it seems to work.  Just need to craft a Proustian comment message for Herr Commandant.

PetscErrorCode SNESSetWorkVecs(SNES snes,PetscInt nw)
{
  PetscErrorCode ierr;
  DM             dm;

  PetscFunctionBegin;
  if (snes->work) {ierr = VecDestroyVecs(snes->nwork,&snes->work);CHKERRQ(ierr);}
  snes->nwork = nw;

  ierr = SNESGetDM(snes,&dm);CHKERRQ(ierr);
  if (dm){
    Vec vec;
    ierr = DMCreateGlobalVector(dm,&vec);CHKERRQ(ierr);
    ierr = VecDuplicateVecs(vec,snes->nwork,&snes->work);CHKERRQ(ierr);
    ierr = VecDestroy(&vec);CHKERRQ(ierr);
  }
  else if (snes->vec_sol) {
    ierr = VecDuplicateVecs(snes->vec_sol,snes->nwork,&snes->work);CHKERRQ(ierr);
  }
  else {
    SETERRQ(PetscObjectComm((PetscObject)snes),PETSC_ERR_SUP,"Can not create work vectors.");
  }

  ierr = PetscLogObjectParents(snes,nw,snes->work);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}


On Oct 11, 2013, at 2:11 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 10, 2013 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On Oct 10, 2013, at 2:37 PM, "Mark F. Adams" <mfadams at lbl.gov> wrote:
> 
> >>
> >> That's what I suggested below.  I can test this now if you like.
> >>
> >> Excellent. I just pushed to next.
> >
> > It is almost done but SNESSetWorkVecs tries to use vec_sol to create work vectors.
> >
> 
>    If SNES has a DM then it should be trying to get its work vectors from that.
> 
> Now a SNES always has a DM. Changed.
> 
>   Matt
>  
> 
>    Barry
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/b291414f/attachment.html>

From knepley at gmail.com  Fri Oct 11 15:00:16 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 Oct 2013 15:00:16 -0500
Subject: [petsc-users] SNESSetUp
In-Reply-To: <B03C44F8-EBC4-4537-8078-771D12511738@lbl.gov>
References: <455B0636-A860-4F8E-96E5-1A8F63954804@lbl.gov>
	<CAMYG4GmN8F0vKd0cH90=A6RW6ekmG8evXC+=puZmQK3QHzE7cA@mail.gmail.com>
	<3C7719EE-9C4B-4C18-988F-F974E5CB7298@lbl.gov>
	<CAMYG4G=jRKYbvO=W_=bKso8NYptH7YHp2hwnH47KewyAEt7R5w@mail.gmail.com>
	<48AA4946-795A-443B-B3B1-F34FF8362859@lbl.gov>
	<CAMYG4Gn2T51b3C3uyPNc+zQLwAHU=V5WiHSpoiLTQ2PaXiZmJQ@mail.gmail.com>
	<760E2724-A18B-447F-AED5-9AB45A24C16E@lbl.gov>
	<38980DE7-CF48-4782-A6E7-218A25E3EF68@mcs.anl.gov>
	<CAMYG4GmgAcpJOoZGhEubApQ6PN1qGLyHM4WKvLXfABV1zUhcLg@mail.gmail.com>
	<B03C44F8-EBC4-4537-8078-771D12511738@lbl.gov>
Message-ID: <CAMYG4Gkvzh0Cd2+A6OYd0+tGJM5HVfc_+QJyN-hzQW3Obf5w-A@mail.gmail.com>

On Fri, Oct 11, 2013 at 2:59 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

> I am working on fixing this in Newton, getting the DM.  I've tested and it
> seems to work.  Just need to craft a Proustian comment message for Herr
> Commandant.
>

Mark, this is already fixed. I pushed it.

  Matt


> PetscErrorCode SNESSetWorkVecs(SNES snes,PetscInt nw)
> {
>   PetscErrorCode ierr;
>   DM             dm;
>
>   PetscFunctionBegin;
>   if (snes->work) {ierr =
> VecDestroyVecs(snes->nwork,&snes->work);CHKERRQ(ierr);}
>   snes->nwork = nw;
>
>   ierr = SNESGetDM(snes,&dm);CHKERRQ(ierr);
>   if (dm){
>     Vec vec;
>     ierr = DMCreateGlobalVector(dm,&vec);CHKERRQ(ierr);
>     ierr = VecDuplicateVecs(vec,snes->nwork,&snes->work);CHKERRQ(ierr);
>     ierr = VecDestroy(&vec);CHKERRQ(ierr);
>   }
>   else if (snes->vec_sol) {
>     ierr =
> VecDuplicateVecs(snes->vec_sol,snes->nwork,&snes->work);CHKERRQ(ierr);
>   }
>   else {
>     SETERRQ(PetscObjectComm((PetscObject)snes),PETSC_ERR_SUP,"Can not
> create work vectors.");
>   }
>
>   ierr = PetscLogObjectParents(snes,nw,snes->work);CHKERRQ(ierr);
>   PetscFunctionReturn(0);
> }
>
>
> On Oct 11, 2013, at 2:11 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Oct 10, 2013 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> On Oct 10, 2013, at 2:37 PM, "Mark F. Adams" <mfadams at lbl.gov> wrote:
>>
>> >>
>> >> That's what I suggested below.  I can test this now if you like.
>> >>
>> >> Excellent. I just pushed to next.
>> >
>> > It is almost done but SNESSetWorkVecs tries to use vec_sol to create
>> work vectors.
>> >
>>
>>    If SNES has a DM then it should be trying to get its work vectors from
>> that.
>
>
> Now a SNES always has a DM. Changed.
>
>   Matt
>
>
>>
>>    Barry
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131011/8d54fb26/attachment.html>

From irving at naml.us  Sat Oct 12 18:13:28 2013
From: irving at naml.us (Geoffrey Irving)
Date: Sat, 12 Oct 2013 16:13:28 -0700
Subject: [petsc-users] [PATCH] I figure someone will appreciate this
Message-ID: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>

Least important patch ever?

Geoffrey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131012/dd7d8fae/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-makefile-Fix-number-of-stars.patch
Type: application/octet-stream
Size: 1326 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131012/dd7d8fae/attachment.obj>

From jbermeol at purdue.edu  Sun Oct 13 12:18:43 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Sun, 13 Oct 2013 13:18:43 -0400 (EDT)
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <635244856.233835.1381115028883.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <39824637.258159.1381684723946.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi again, so I haven't complete the mkl-cpardiso interface for some compilation problems. However I did an interfaces for the regular pardiso that comes with intel-mkl. Now there are a couple of things pending to do before to do the pool request:

1. I did a mkl-pardiso.py script for the installation of the solver. It checks that BlasLapack is installed, but it doesn't check that the BlasLapack implementation is mkl implementation. I did't know how to do that so maybe you cold helpme with that.

2. Intel suggest to use the following flags for the linker phase "-Wl,--start-group -L(PATH_TO_MKL) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5". This is simple if I use -with-mkl-pardiso-lib. But if I want to use -with-mkl-pardiso-dir is more complicated and need to be done.

3. Finally related with the -with-mkl-pardiso-dir I don't if would be necessary to do a library validation, apart from the BlasLapack validation.

If you could help me with this question, I would be able to do a pull request soon.

Thanks   



----- Original Message -----
From: "Jose David Bermeol" <jbermeol at purdue.edu>
To: "Barry Smith" <bsmith at mcs.anl.gov>
Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 11:03:49 PM
Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration

Yes, blaslapack.py is enough big and horrible to add more code. Well the easier configuration is not a high priority think for me(but it something that in some point I will have to do). So for now I will have a basic cpardiso.py and then I will try to ensure that the blas/lapack libraries are correct.

----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
Sent: Sunday, October 6, 2013 10:54:53 PM
Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration


   Jose,

    Since you are currently working on interfacing cpardiso I would just have a cpardiso.py (or cpardiso_mkl.py) (and NO mkl.py) that manages/lists the libraries needed for cpardiso. Don't have both a cpardiso.py and cpardiso_mkl.py. 

    Since, as Satish reports, cpardiso.py brings in also the mkl blas/lapack I would also have the cpardiso.py script check that the blas/lapack libraries being linked are compatible with the cpardiso mkl libraries being brought in and provide a useful error message telling them how to set the blas/lapack libraries to fix the problem when needed.

   We definitely do not want to muddy up the already horrible blaslapack.py code by introducing the concept of cpardiso there.

    Anyways, once you've simplified things, just make a pull request at bitbucket and we can try it and help with the kinks.


   Barry



On Oct 6, 2013, at 9:07 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  
> 
> ----- Original Message -----
> From: "Satish Balay" <balay at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 9:57:50 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
> etc - then things will break.
> 
> So somehow these two options should work together - and perhaps the
> way they can do is within blaslapack.py [and not a separate mkl.py]
> 
> Satish
> 
> On Sun, 6 Oct 2013, Jose David Bermeol wrote:
> 
>> Exactly
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "Matthew Knepley" <knepley at gmail.com>
>> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Sunday, October 6, 2013 9:49:59 PM
>> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
>> 
>> 
>>   Because he is using other stuff in mkl besides blas/lapack!
>> 
>> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
>> 
>>> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
>>> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
>>> 
>>> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
>>> 
>>> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
>> 
>>   Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
>> 
>>   Barry
>> 
>>> 
>>> Is there a reason you do not want to use --with-blas-lapack-lib?
>>> 
>>>  Thanks,
>>> 
>>>    Matt
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 

From bsmith at mcs.anl.gov  Sun Oct 13 13:09:42 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 13 Oct 2013 13:09:42 -0500
Subject: [petsc-users] [petsc-dev]  mkl-cpardiso configuration
In-Reply-To: <39824637.258159.1381684723946.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <39824637.258159.1381684723946.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <816D41F6-685E-4280-A33F-24EE4D7B1394@mcs.anl.gov>


On Oct 13, 2013, at 12:18 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:

> Hi again, so I haven't complete the mkl-cpardiso interface for some compilation problems. However I did an interfaces for the regular pardiso that comes with intel-mkl. Now there are a couple of things pending to do before to do the pool request:
> 
> 1. I did a mkl-pardiso.py script for the installation of the solver. It checks that BlasLapack is installed, but it doesn't check that the BlasLapack implementation is mkl implementation. I did't know how to do that so maybe you cold helpme with that.
> 
> 2. Intel suggest to use the following flags for the linker phase "-Wl,--start-group -L(PATH_TO_MKL) -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5". This is simple if I use -with-mkl-pardiso-lib. But if I want to use -with-mkl-pardiso-dir is more complicated and need to be done.

   With -with-mkl-pardiso-dir   you would just list that monster as the first guess, it is trivial. With -with-mkl-pardiso-lib the user needs to type in that monster, that is just the way it is.

> 
> 3. Finally related with the -with-mkl-pardiso-dir I don't if would be necessary to do a library validation, apart from the BlasLapack validation.

    You always need to make sure that the library is likely to work, this means (like with all the other packages) you link against some symbol you know is in the library to make sure it is there.

   Barry

> 
> If you could help me with this question, I would be able to do a pull request soon.
> 
> Thanks   
> 
> 
> 
> ----- Original Message -----
> From: "Jose David Bermeol" <jbermeol at purdue.edu>
> To: "Barry Smith" <bsmith at mcs.anl.gov>
> Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 11:03:49 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> Yes, blaslapack.py is enough big and horrible to add more code. Well the easier configuration is not a high priority think for me(but it something that in some point I will have to do). So for now I will have a basic cpardiso.py and then I will try to ensure that the blas/lapack libraries are correct.
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
> Sent: Sunday, October 6, 2013 10:54:53 PM
> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
> 
> 
>   Jose,
> 
>    Since you are currently working on interfacing cpardiso I would just have a cpardiso.py (or cpardiso_mkl.py) (and NO mkl.py) that manages/lists the libraries needed for cpardiso. Don't have both a cpardiso.py and cpardiso_mkl.py. 
> 
>    Since, as Satish reports, cpardiso.py brings in also the mkl blas/lapack I would also have the cpardiso.py script check that the blas/lapack libraries being linked are compatible with the cpardiso mkl libraries being brought in and provide a useful error message telling them how to set the blas/lapack libraries to fix the problem when needed.
> 
>   We definitely do not want to muddy up the already horrible blaslapack.py code by introducing the concept of cpardiso there.
> 
>    Anyways, once you've simplified things, just make a pull request at bitbucket and we can try it and help with the kinks.
> 
> 
>   Barry
> 
> 
> 
> On Oct 6, 2013, at 9:07 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
> 
>> Well actually I should change the name of the file mkl.py to mkl_cpardiso.py. It doesn't pick up any blas/lapack library. So it would be save and convenient to have it as a separate package(just to keep things modular and simple). If in a near future the solver would need a blas/lapack library I would agree that the blaslapack.py should take care of the installation of the solver.  
>> 
>> ----- Original Message -----
>> From: "Satish Balay" <balay at mcs.anl.gov>
>> To: "Jose David Bermeol" <jbermeol at purdue.edu>
>> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>> Sent: Sunday, October 6, 2013 9:57:50 PM
>> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
>> 
>> If one specifies '--with-mkl=/foo/bar --download-f-blas-lapack' or
>> just '--with-mlk=/foo/bar' where configure picks up /usr/lib/libblas.a
>> etc - then things will break.
>> 
>> So somehow these two options should work together - and perhaps the
>> way they can do is within blaslapack.py [and not a separate mkl.py]
>> 
>> Satish
>> 
>> On Sun, 6 Oct 2013, Jose David Bermeol wrote:
>> 
>>> Exactly
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "Matthew Knepley" <knepley at gmail.com>
>>> Cc: "Jose David Bermeol" <jbermeol at purdue.edu>, petsc-users at mcs.anl.gov, "For users of the development version of PETSc" <petsc-dev at mcs.anl.gov>
>>> Sent: Sunday, October 6, 2013 9:49:59 PM
>>> Subject: Re: [petsc-dev] [petsc-users] mkl-cpardiso configuration
>>> 
>>> 
>>>  Because he is using other stuff in mkl besides blas/lapack!
>>> 
>>> On Oct 6, 2013, at 8:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> 
>>>> On Sun, Oct 6, 2013 at 8:04 PM, Jose David Bermeol <jbermeol at purdue.edu> wrote:
>>>> Hi, one question about how to configure a new package, intel people suggest to compile mkl-cpardiso using the following flags for the linker:
>>>> 
>>>> -Wl,--start-group -L$(MKL_PATH)/lib/intel64/ -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -lpthread -lm -liomp5
>>>> 
>>>> Now I want to be able to do this just using the flag --with-mkl-dir=$(MKL_PATH) during Petsc configuration. I've been trying to modify my mkl.py file to achieve this, however your configuration system is a little bit complex, so if you code give some guidelines to achieve this or files to look how to do it would be great.
>>> 
>>>  Make a pull request with all your current stuff and we can look at how you've organized it and make suggestions. Otherwise it is too much email guess work with incomplete information.
>>> 
>>>  Barry
>>> 
>>>> 
>>>> Is there a reason you do not want to use --with-blas-lapack-lib?
>>>> 
>>>> Thanks,
>>>> 
>>>>   Matt
>>>> 
>>>> Thanks
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 


From rupp at mcs.anl.gov  Sun Oct 13 15:14:45 2013
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sun, 13 Oct 2013 22:14:45 +0200
Subject: [petsc-users] [PATCH] I figure someone will appreciate this
In-Reply-To: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
Message-ID: <525AFF35.7020501@mcs.anl.gov>

Hi Geoffrey,

I like it - could you please provide a good commit message, cf.
https://bitbucket.org/pmullowney/petsc-gpu-dev/commits/f68d8d3433d7022f2153a4f14dea626b439a8406
? ;-)

Moreover, is there any reason for the new number of stars? I can't see 
the pattern, particularly as there are 26 stars before 'ERROR' and 37 
stars after...

Best regards,
Karli


On 10/13/2013 01:13 AM, Geoffrey Irving wrote:
> Least important patch ever?
>
> Geoffrey
>
>


From irving at naml.us  Sun Oct 13 15:28:36 2013
From: irving at naml.us (Geoffrey Irving)
Date: Sun, 13 Oct 2013 13:28:36 -0700
Subject: [petsc-users] [PATCH] I figure someone will appreciate this
In-Reply-To: <525AFF35.7020501@mcs.anl.gov>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
Message-ID: <CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>

I think Jed already pulled it, but sorry for the lack of explanation.  With
this patch, the line lengths of the two lines with stars match.

Geoffrey


On Sun, Oct 13, 2013 at 1:14 PM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Geoffrey,
>
> I like it - could you please provide a good commit message, cf.
> https://bitbucket.org/**pmullowney/petsc-gpu-dev/**commits/**
> f68d8d3433d7022f2153a4f14dea62**6b439a8406<https://bitbucket.org/pmullowney/petsc-gpu-dev/commits/f68d8d3433d7022f2153a4f14dea626b439a8406>
> ? ;-)
>
> Moreover, is there any reason for the new number of stars? I can't see the
> pattern, particularly as there are 26 stars before 'ERROR' and 37 stars
> after...
>
> Best regards,
> Karli
>
>
>
> On 10/13/2013 01:13 AM, Geoffrey Irving wrote:
>
>> Least important patch ever?
>>
>> Geoffrey
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131013/86d0f0d4/attachment.html>

From jedbrown at mcs.anl.gov  Sun Oct 13 15:43:13 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 13 Oct 2013 15:43:13 -0500
Subject: [petsc-users] [PATCH] I figure someone will appreciate this
In-Reply-To: <CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
	<CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
Message-ID: <87fvs4ucke.fsf@mcs.anl.gov>

Geoffrey Irving <irving at naml.us> writes:

> I think Jed already pulled it, but sorry for the lack of explanation.  With
> this patch, the line lengths of the two lines with stars match.

I haven't pushed it yet, but I was going to amend the commit message to
say the above.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131013/ad9e8122/attachment.pgp>

From rupp at mcs.anl.gov  Sun Oct 13 15:44:20 2013
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Sun, 13 Oct 2013 22:44:20 +0200
Subject: [petsc-users] [PATCH] I figure someone will appreciate this
In-Reply-To: <CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
	<CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
Message-ID: <525B0624.9060409@mcs.anl.gov>

Hey,

 > I think Jed already pulled it, but sorry for the lack of explanation.
> With this patch, the line lengths of the two lines with stars match.

thanks - in such case I'll leave it to Jed...

Best regards,
Karli

From popov at uni-mainz.de  Mon Oct 14 04:10:12 2013
From: popov at uni-mainz.de (Anton Popov)
Date: Mon, 14 Oct 2013 11:10:12 +0200
Subject: [petsc-users] matnest
In-Reply-To: <525B0624.9060409@mcs.anl.gov>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
	<CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
	<525B0624.9060409@mcs.anl.gov>
Message-ID: <525BB4F4.2020704@uni-mainz.de>

Hi all,

do I need to call MatAssemblyBegin/End on a MATNEST matrix again after 
the submatrices get updated by MatSetValues + MatAssemblyBegin/End?

Thanks,

Anton



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131014/24c85c86/attachment-0001.html>

From dave.mayhem23 at gmail.com  Mon Oct 14 04:47:32 2013
From: dave.mayhem23 at gmail.com (Dave May)
Date: Mon, 14 Oct 2013 11:47:32 +0200
Subject: [petsc-users] matnest
In-Reply-To: <525BB4F4.2020704@uni-mainz.de>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
	<CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
	<525B0624.9060409@mcs.anl.gov> <525BB4F4.2020704@uni-mainz.de>
Message-ID: <CAJ98EDrsH7wyHma7orYW6mnDrkNWUmQ19LfebSRv2aOuNfBgVw@mail.gmail.com>

Yes

On Monday, 14 October 2013, Anton Popov wrote:

>  Hi all,
>
> do I need to call MatAssemblyBegin/End on a MATNEST matrix again after
> the  submatrices get updated by MatSetValues + MatAssemblyBegin/End?
>
> Thanks,
>
> Anton
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131014/a0c3868f/attachment.html>

From popov at uni-mainz.de  Tue Oct 15 04:35:07 2013
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 15 Oct 2013 11:35:07 +0200
Subject: [petsc-users] matnest
In-Reply-To: <CAJ98EDrsH7wyHma7orYW6mnDrkNWUmQ19LfebSRv2aOuNfBgVw@mail.gmail.com>
References: <CAJ1ofpda65ySyhw8=xJzE0bQNuuxtdYSVcGc5P7tkuc8cGiDFA@mail.gmail.com>
	<525AFF35.7020501@mcs.anl.gov>
	<CAJ1ofpfn8Uagd0pkYTT1Bon3Wcpv162ZedePxg+OWpkR4Tn_QQ@mail.gmail.com>
	<525B0624.9060409@mcs.anl.gov> <525BB4F4.2020704@uni-mainz.de>
	<CAJ98EDrsH7wyHma7orYW6mnDrkNWUmQ19LfebSRv2aOuNfBgVw@mail.gmail.com>
Message-ID: <525D0C4B.8020407@uni-mainz.de>

On 10/14/13 11:47 AM, Dave May wrote:
> Yes
Thanks Dave
>
> On Monday, 14 October 2013, Anton Popov wrote:
>
>     Hi all,
>
>     do I need to call MatAssemblyBegin/End on a MATNEST matrix again
>     after the  submatrices get updated by MatSetValues +
>     MatAssemblyBegin/End?
>
>     Thanks,
>
>     Anton
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/a3a890a9/attachment.html>

From christophe.ortiz at ciemat.es  Tue Oct 15 07:29:22 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 15 Oct 2013 14:29:22 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
Message-ID: <CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>

Hi all,

I'm still working on my code with dof>>1 and I came to the point where I
want to use DMDASetBlcokFills to set the pattern of each block (see ex10.c
from advection-diffusion).

I have a doubt...
Let's imagine I have a system of 2 diffusion equations (dof=2) with one
coupling, both with Dirichlet boundary conditions:

u_t - D u_xx = -k*u*v        u(x=0,t)=u(x=L,t)=0
v_t - D v_xx = -k*u*v         v(x=0,t)=v(x=L,t)=0

If I write everything under IFunction, then the IJacobian is:

       1        0          |
       0        1          |

-D/dx^2   0           | a+2D/dx^2+kv       +ku  | -D/dx^2     0
    0        -D/dx^2  |         +kv                  +ku  |      0
-D/dx^2


The first block on top represents Dirichlet boundary conditions. The three
other blocks are for interior nodes.

My question is on how to set dfill matrix. For boundary conditions, the
diagonal block does not have the same pattern as the one for interior node.
We see that for node i=0, components are just coupled with themselves while
for the interior diagonal block, they are coupled one with each other.
Then, how to set dfill ? Should it be

{1,   0,
 0,    1 }

or

{ 1,   1,
  1,    1}   ?

BTW, does DMDASetBlockFills help during solving or is it just for managing
memory ?

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/f8209855/attachment.html>

From jedbrown at mcs.anl.gov  Tue Oct 15 07:53:32 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 Oct 2013 07:53:32 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
Message-ID: <878uxur8z7.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> I have a doubt...
> Let's imagine I have a system of 2 diffusion equations (dof=2) with one
> coupling, both with Dirichlet boundary conditions:
>
> u_t - D u_xx = -k*u*v        u(x=0,t)=u(x=L,t)=0
> v_t - D v_xx = -k*u*v         v(x=0,t)=v(x=L,t)=0
>
> If I write everything under IFunction, then the IJacobian is:
>
>        1        0          |
>        0        1          |
>
> -D/dx^2   0           | a+2D/dx^2+kv       +ku  | -D/dx^2     0
>     0        -D/dx^2  |         +kv                  +ku  |      0
> -D/dx^2
>
>
> The first block on top represents Dirichlet boundary conditions. The three
> other blocks are for interior nodes.
>
> My question is on how to set dfill matrix. For boundary conditions, the
> diagonal block does not have the same pattern as the one for interior node.
> We see that for node i=0, components are just coupled with themselves while
> for the interior diagonal block, they are coupled one with each other.
> Then, how to set dfill ? Should it be
>
> {1,   0,
>  0,    1 }
>
> or
>
> { 1,   1,
>   1,    1}   ?

You have to preallocate for the max number of entries in each kind of
block.  The slight bit of extra memory for the boundary conditions will
be squeezed out and not affect performance.

> BTW, does DMDASetBlockFills help during solving or is it just for managing
> memory ?

Allocated nonzeros that have never been "set" are squeezed out in
assembly, so they only affect memory.  Allocated entries that are
explicitly set to 0.0 are not squeezed out, though there is a Mat option
to do that.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/d584568d/attachment.pgp>

From christophe.ortiz at ciemat.es  Tue Oct 15 09:15:30 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 15 Oct 2013 16:15:30 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <878uxur8z7.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<878uxur8z7.fsf@mcs.anl.gov>
Message-ID: <CANBrw+6FKTbLiL_uPm0igqTp1hYjTOO3BZiLgfRSwCFY9-6N8w@mail.gmail.com>

On Tue, Oct 15, 2013 at 2:53 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > I have a doubt...
> > Let's imagine I have a system of 2 diffusion equations (dof=2) with one
> > coupling, both with Dirichlet boundary conditions:
> >
> > u_t - D u_xx = -k*u*v        u(x=0,t)=u(x=L,t)=0
> > v_t - D v_xx = -k*u*v         v(x=0,t)=v(x=L,t)=0
> >
> > If I write everything under IFunction, then the IJacobian is:
> >
> >        1        0          |
> >        0        1          |
> >
> > -D/dx^2   0           | a+2D/dx^2+kv       +ku  | -D/dx^2     0
> >     0        -D/dx^2  |         +kv                  +ku  |      0
> > -D/dx^2
> >
> >
> > The first block on top represents Dirichlet boundary conditions. The
> three
> > other blocks are for interior nodes.
> >
> > My question is on how to set dfill matrix. For boundary conditions, the
> > diagonal block does not have the same pattern as the one for interior
> node.
> > We see that for node i=0, components are just coupled with themselves
> while
> > for the interior diagonal block, they are coupled one with each other.
> > Then, how to set dfill ? Should it be
> >
> > {1,   0,
> >  0,    1 }
> >
> > or
> >
> > { 1,   1,
> >   1,    1}   ?
>
> You have to preallocate for the max number of entries in each kind of
> block.  The slight bit of extra memory for the boundary conditions will
> be squeezed out and not affect performance.
>

Thanks. Ok, so I guess that I should use {1, 1, 1, 1} to preallocate for
the max number of entries.


>
> > BTW, does DMDASetBlockFills help during solving or is it just for
> managing
> > memory ?
>
> Allocated nonzeros that have never been "set" are squeezed out in
> assembly, so they only affect memory.  Allocated entries that are
> explicitly set to 0.0 are not squeezed out, though there is a Mat option
> to do that.
>

I see.

Before I go on implementing all the coupling between all the dof, I am
doing some tests with the system described above and I see some strange
behavior...
To do some tests, as initial conditions I set two gaussian distributions
for u and v. When the peak value of the gaussians is in the order of unity,
no problem, ARKIMEX as well as ROSW work fine. However, when I use peak
values in the order of 1e15-1e20 (which is of interest for me), then
troubles arise. I see it takes much more time to do one timestep for
ARKIMEX. This, even when I switch off the coupling and there is only
diffusion. It takes too long.
 I tried KSPCG and KSPGMRES and PCNONE/PCILU.

- is it due to some problem with ARKIMEX and large values ?

- How do you use ts_adapt_basic_clip in the run command to decrease the
fastest increase of timestep ? I would like to set 1.5 for the fastest
increase instead of 10, which is the default. I tried but PETSc complains
saying:

[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: Must give exactly two values to -ts_adapt_basic_clip!

- ARKIMEX seems to work better if I decrease the number of nodes, ie if I
increase the dx.

- I found out with ts_view that the ARKIMEX type is always ARKIMEX3, even
when I use

   ierr = TSARKIMEXSetType(ts,TSARKIMEX1BEE);

Is it normal ?

- With TSROSW, much better. Any idea why ?

- Do you recommend a particular KSP/PC for this type of problem ?


Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/d77e7743/attachment.html>

From jedbrown at mcs.anl.gov  Tue Oct 15 09:36:55 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 Oct 2013 09:36:55 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6FKTbLiL_uPm0igqTp1hYjTOO3BZiLgfRSwCFY9-6N8w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<878uxur8z7.fsf@mcs.anl.go
	v> <CANBrw+6FKTbLiL_uPm0igqTp1hYjTOO3BZiLgfRSwCFY9-6N8w@mail.gmail.com>
Message-ID: <8761syr46w.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> Before I go on implementing all the coupling between all the dof, I am
> doing some tests with the system described above and I see some strange
> behavior...
> To do some tests, as initial conditions I set two gaussian distributions
> for u and v. When the peak value of the gaussians is in the order of unity,
> no problem, ARKIMEX as well as ROSW work fine. However, when I use peak
> values in the order of 1e15-1e20 (which is of interest for me), then
> troubles arise. I see it takes much more time to do one timestep for
> ARKIMEX. This, even when I switch off the coupling and there is only
> diffusion. It takes too long.
>  I tried KSPCG and KSPGMRES and PCNONE/PCILU.
>
> - is it due to some problem with ARKIMEX and large values ?

Error control should be the same.  Run with -ts_adapt_monitor and send
output if you don't understand it.

> - How do you use ts_adapt_basic_clip in the run command to decrease the
> fastest increase of timestep ? I would like to set 1.5 for the fastest
> increase instead of 10, which is the default. I tried but PETSc complains
> saying:
>
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: Must give exactly two values to -ts_adapt_basic_clip!

-ts_adapt_basic_clip 0.1,1.5

The first number is the fastest decrease that will be allowed.

> - ARKIMEX seems to work better if I decrease the number of nodes, ie if I
> increase the dx.

Then you'll be swamped with spatial error.

> - I found out with ts_view that the ARKIMEX type is always ARKIMEX3, even
> when I use
>
>    ierr = TSARKIMEXSetType(ts,TSARKIMEX1BEE);

Maybe you call this before the type has been set, e.g., before
TSSetFromOptions().  Use -ts_arkimex_type 1bee.

> - With TSROSW, much better. Any idea why ?

Default TSROSW method (ra34pw2) has a good error estimate.  It's
probably faster because it only does linear solves.  If your problem is
linear (or weakly nonlinear), TSROSW is usually better than ARKIMEX, at
least unless you tell the SNES to compute the Jacobian only once.

> - Do you recommend a particular KSP/PC for this type of problem ?

Use a direct solver until you understand what is happening in the outer
levels.  Then use multigrid because it is the right way to solve the problem.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/86db03ef/attachment-0001.pgp>

From christophe.ortiz at ciemat.es  Tue Oct 15 10:40:35 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 15 Oct 2013 17:40:35 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <8761syr46w.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+6FKTbLiL_uPm0igqTp1hYjTOO3BZiLgfRSwCFY9-6N8w@mail.gmail.com>
	<8761syr46w.fsf@mcs.anl.gov>
Message-ID: <CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>

On Tue, Oct 15, 2013 at 4:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > Before I go on implementing all the coupling between all the dof, I am
> > doing some tests with the system described above and I see some strange
> > behavior...
> > To do some tests, as initial conditions I set two gaussian distributions
> > for u and v. When the peak value of the gaussians is in the order of
> unity,
> > no problem, ARKIMEX as well as ROSW work fine. However, when I use peak
> > values in the order of 1e15-1e20 (which is of interest for me), then
> > troubles arise. I see it takes much more time to do one timestep for
> > ARKIMEX. This, even when I switch off the coupling and there is only
> > diffusion. It takes too long.
> >  I tried KSPCG and KSPGMRES and PCNONE/PCILU.
> >
> > - is it due to some problem with ARKIMEX and large values ?
>
> Error control should be the same.  Run with -ts_adapt_monitor and send
> output if you don't understand it.
>
> > - How do you use ts_adapt_basic_clip in the run command to decrease the
> > fastest increase of timestep ? I would like to set 1.5 for the fastest
> > increase instead of 10, which is the default. I tried but PETSc complains
> > saying:
> >
> > [0]PETSC ERROR: Argument out of range!
> > [0]PETSC ERROR: Must give exactly two values to -ts_adapt_basic_clip!
>
> -ts_adapt_basic_clip 0.1,1.5
>
> The first number is the fastest decrease that will be allowed.
>

Great, it works now.


>
> > - ARKIMEX seems to work better if I decrease the number of nodes, ie if I
> > increase the dx.
>
> Then you'll be swamped with spatial error.
>
> > - I found out with ts_view that the ARKIMEX type is always ARKIMEX3, even
> > when I use
> >
> >    ierr = TSARKIMEXSetType(ts,TSARKIMEX1BEE);
>
> Maybe you call this before the type has been set, e.g., before
> TSSetFromOptions().  Use -ts_arkimex_type 1bee.
>

Yes ! Actually, that was the problem ! Now I set ARKIMEX1BEE after
TSSetFromOptions() and it is much faster ! ARKIMEXA2 seems even faster.
Thanks Jed !

Now I see that the convergence speed of ARKIMEX depends on where I put
TSSetFromOptions().

If I do:
TSCreate();
TSSetFromOptions();
TSSetProblemType();
TSSetType();
TSARKIMEXSetType();

then ARKIMEXA2 is very fast.

Instead, if I put TSSetFromOptions() somewhere else:
TSCreate();

TSSetProblemType();
TSSetType();
*TSSetFromOptions();*
TSARKIMEXSetType();

ARKIMEXA2 slows down significantly. I guess that TSSetFromOptions overrides
something...Any idea ?


>
> > - With TSROSW, much better. Any idea why ?
>
> Default TSROSW method (ra34pw2) has a good error estimate.  It's
> probably faster because it only does linear solves.  If your problem is
> linear (or weakly nonlinear), TSROSW is usually better than ARKIMEX, at
> least unless you tell the SNES to compute the Jacobian only once.
>

Good to know.


>
> > - Do you recommend a particular KSP/PC for this type of problem ?
>
> Use a direct solver until you understand what is happening in the outer
> levels.  Then use multigrid because it is the right way to solve the
> problem.
>

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/9ed1b142/attachment.html>

From jedbrown at mcs.anl.gov  Tue Oct 15 11:20:08 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 Oct 2013 11:20:08 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+6FKTbLiL_uPm0igqTp
	1hYjTOO3BZiLgfRSwCFY9-6N8w@mail.gmail.com> <8761syr46w.fsf@mcs.anl.gov>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
Message-ID: <87k3hepkuf.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> Yes ! Actually, that was the problem ! Now I set ARKIMEX1BEE after
> TSSetFromOptions() and it is much faster ! ARKIMEXA2 seems even faster.

I still recommend configuring this stuff from the command line, and I
would put TSSetFromOptions last so that you can override anything at
run-time.

> Thanks Jed !
>
> Now I see that the convergence speed of ARKIMEX depends on where I put
> TSSetFromOptions().
>
> If I do:
> TSCreate();
> TSSetFromOptions();
> TSSetProblemType();
> TSSetType();
> TSARKIMEXSetType();
>
> then ARKIMEXA2 is very fast.
>
> Instead, if I put TSSetFromOptions() somewhere else:
> TSCreate();
>
> TSSetProblemType();
> TSSetType();
> *TSSetFromOptions();*
> TSARKIMEXSetType();

I would use this order:

    TSCreate();
    TSSetProblemType();  // optional
    TSSetType();
    TSARKIMEXSetType();
    TSSetFromOptions();

> ARKIMEXA2 slows down significantly. I guess that TSSetFromOptions overrides
> something...Any idea ?

Compare the output from -ts_view in both cases.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/3600ec5a/attachment.pgp>

From balay at mcs.anl.gov  Tue Oct 15 13:29:40 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 15 Oct 2013 13:29:40 -0500 (CDT)
Subject: [petsc-users] petsc-3.4.3.tar.gz now available
Message-ID: <alpine.LFD.2.03.1310151318220.2161@mcs.anl.gov>

Dear PETSc users,

The patch release petsc-3.4.3 is now available for download.
http://www.mcs.anl.gov/petsc/download/index.html

Some of the changes include:

    * Sys: Fix C/C++ choice for PETSC_RESTRICT and PETSC_STATIC_INLINE.

    * PCGAMG: fix corruption for multiple solves when MatSetNearNullSpace is used

    * Mat: add MATOP_ values for missing functions

    * Intel: Fix error encountered when using Intel compiler and changing gcc on
      PATH between configuring PETSc and building an application.

    * Configure: avoid string argument to __attribute((deprecated))

    * KSPDGMRES: fix conditional variable declarations

    * MatSBAIJ: avoid invalid memory read in MatSOR_SBAIJ

    * BGQ: Work around BG/Q MPI_Bcast bug.

    * PetscOptionsInsertFile: combine Bcasts for alias and values

    * Sys: ex13 avoid non-portable realpath(1) in verification test

    * Sys: fix PetscGetFullPath expansion of ~ and ~user, ensure NULL termination

    * Sys: avoid getpwuid in file path routines

    * MatXAIJSetPreallocation: use array[] notation to help out Fortran

    * Cygwin64: Fix build error with fenv.h

    * MatSchurComplement: Must initialize KSP package

    * SNES: Fix accidental creation of the matrices in SNESNCG used for linear analogue LS.

    * PCGAMG: add man page for PCGAMGType

    * SNES: Fixed SNESComputeJacobianDefaultColor() in the case of multilevel PC

    * Mat: fix Fortran interface for MatGetColumnNorms

    * Bfort: distinguishes between array[] and *pointer to decide whether the
      Fortran interface should have an array or a scalar.

    * DM: Fix for accidentally duplicated refcount decrement in dm->x

    * Doc: PetscLayoutCreate format code block and list

    * Sys: add PetscDataTypeFromString() and test code

    * PCGAMG: Always rebuild coarse Mat the first time

    * PCFactor: add PCFactorGetLevels

    * Dgmres: added option to view deflation vecs after solve.

    * MatPCFactor: include MatFactorShiftType in PCView_Factor output

    * AO: fix erroneous processing of -ao_view and factor into AOViewFromOptions

    * Viewer ASCII: print tabs to petsc_history, not fd

    * PCBJacobi: call KSPSetFromOptions(subksp) when PCSetUp was called first

    * PCGAMG: set bjacobi->same_local_solves=TRUE to clean coarse grid viewing

    * SNESMonitorCancel: enable fortran interface

    * BGQ: fix deadlock on bgq due to inconsistant values returned by getpwuid(getuid())

    * Scalapack: fix build by specifying mpi.h to build

    * MatSeqAIJGetArrayF90: add man pages and fix typo in MatDense version

    * SF docs: add link to summary page and fix title

    * Doc: Fixed DM code in manual

    * VTK: Small fix to error message (.vts to .vtu)

    * Config cmakeboot: add C++ flags any time compiler is available

    * Config OpenMP: check for C++ flag any time the compiler is available

    * Parms: update tarball with fix for namespace conflict with metis

    * Parmetis: update tarball to parmetis-4.0.2-p5 which fixes an install issue with cygwin

    * Viewer: fix Binary Matlab viewer info to use '#' instead of '%'

    * MatSetOption not always logically collective.

    * SNESVI: Fix computation of squared-norm in for the bt linesearch in the VIRS snes case

    * KSPSetDM docs: fix KSPSetDMActive args, add cross-references

    * Vec: fix flop counting to use map->n instead of non-existent map->m

    * Hdf5 installed without zlib should configure without neededing zlib dependency

    * Config: name of configure generated include file confdefs.h did not match name printed in configure.log of conftest.h

    * Vec: put back in flop counts for VecNorm_Seq

    * KSPFGMRES: make KSPMonitor usage match KSPGMRES

    * TS: Tutorial example has no embedded method so needs -ts_adapt_type none
      added MatSetUp calls to TS tutorials 16 and 21

    * DM: Updated DM test ex7.c to use valid command-line options to specify 2D and 3D grids.

    * PCMG: Fixed Fortran binding

    * TS: fix MSVC compiler error: non-POD member of anonymous struct

    * TS tests/ex3: SNESFD and TSComputeIJacobianConstant are incompatible

    * TS: fix shortcut scaling; document TSComputeIJacobianConstant limitations

    * TS: add TSRHSJacobianSetReuse, fixes TSComputeRHSJacobianConstant

    * Zoltan: X is listed as a required dependency of Zoltan but it is only optional

    * Doc: add petsctime.h to SOURCEH (fixes missing docs)

    * ABI: fix ABI compatibility for debug and optimized builds

    * Clang: Fix warning with clang-3.3.

    * Mat Chaco: remove another case of PETSC_DEFAULT out of range of enum

Satish

From jedbrown at mcs.anl.gov  Tue Oct 15 13:37:23 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 Oct 2013 13:37:23 -0500
Subject: [petsc-users] petsc-3.4.3.tar.gz now available
In-Reply-To: <alpine.LFD.2.03.1310151318220.2161@mcs.anl.gov>
References: <alpine.LFD.2.03.1310151318220.2161@mcs.anl.gov>
Message-ID: <87bo2qpeho.fsf@mcs.anl.gov>

Satish Balay <balay at mcs.anl.gov> writes:

> Dear PETSc users,
>
> The patch release petsc-3.4.3 is now available for download.
> http://www.mcs.anl.gov/petsc/download/index.html
>
> Some of the changes include:

To be clear, this is a maintenance release and is both source and binary
compatible with v3.  We encourage everyone to update.  The change log
Satish included are all bug fixes, documentation, or code maintenance
like a better error/warning message.

>     * Sys: Fix C/C++ choice for PETSC_RESTRICT and PETSC_STATIC_INLINE.

Due to this bug fix, you will have to re-run configure after unpacking
the new tarball.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131015/b36eea60/attachment.pgp>

From christophe.ortiz at ciemat.es  Wed Oct 16 07:02:42 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 16 Oct 2013 14:02:42 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87k3hepkuf.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<8761syr46w.fsf@mcs.anl.gov>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
Message-ID: <CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>

On Tue, Oct 15, 2013 at 6:20 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > Yes ! Actually, that was the problem ! Now I set ARKIMEX1BEE after
> > TSSetFromOptions() and it is much faster ! ARKIMEXA2 seems even faster.
>
> I still recommend configuring this stuff from the command line, and I
> would put TSSetFromOptions last so that you can override anything at
> run-time.
>

Ok.


>
> > Thanks Jed !
> >
> > Now I see that the convergence speed of ARKIMEX depends on where I put
> > TSSetFromOptions().
> >
> > If I do:
> > TSCreate();
> > TSSetFromOptions();
> > TSSetProblemType();
> > TSSetType();
> > TSARKIMEXSetType();
> >
> > then ARKIMEXA2 is very fast.
> >
> > Instead, if I put TSSetFromOptions() somewhere else:
> > TSCreate();
> >
> > TSSetProblemType();
> > TSSetType();
> > *TSSetFromOptions();*
> > TSARKIMEXSetType();
>
> I would use this order:
>
>     TSCreate();
>     TSSetProblemType();  // optional
>     TSSetType();
>     TSARKIMEXSetType();
>     TSSetFromOptions();
>
> > ARKIMEXA2 slows down significantly. I guess that TSSetFromOptions
> overrides
> > something...Any idea ?
>
> Compare the output from -ts_view in both cases.
>

I found where is the problem. I had
set TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE);
I found it produces wrong solutions, either with ARKIMEX3, 1BEE or A2.
Solution becomes negative. Without this option, all ARKIMEX types work very
well and very fast.

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/816d5676/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 16 08:36:49 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 08:36:49 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<8761syr46w.fsf@mcs.anl.go
	v> <CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
Message-ID: <87vc0xmj66.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> I found where is the problem. I had
> set TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE);
> I found it produces wrong solutions, either with ARKIMEX3, 1BEE or A2.
> Solution becomes negative. Without this option, all ARKIMEX types work very
> well and very fast.

I'm glad they work as IMEX methods (please check that the solutions are
correct), but I would like to find out why they are not working in fully
implicit mode.  Is your problem similar to ex25?  (If I remember
correctly, that example uses a DAE formulation for boundary conditions
and thus has trouble with some methods with explicit stages.  There may
be more going on.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/ee5041a2/attachment.pgp>

From pengxwang at hotmail.com  Wed Oct 16 09:01:23 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Wed, 16 Oct 2013 09:01:23 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector for a
	3-D domain
Message-ID: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>

Hello,

    I have a DM and its vector for a structured 3-D domain and trying to extract the data for a 2-D plane.  I did some research on the manual and  noticed there is function named DMSlicedCreate(), however, it is for unstructured problem. Is there any ohter function to extract vector for a 2-D plane from DM vector for a 3-D domain directly? Thanks.  		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/1484547f/attachment.html>

From christophe.ortiz at ciemat.es  Wed Oct 16 09:10:43 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 16 Oct 2013 16:10:43 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87vc0xmj66.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
Message-ID: <CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>

On Wed, Oct 16, 2013 at 3:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > I found where is the problem. I had
> > set TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE);
> > I found it produces wrong solutions, either with ARKIMEX3, 1BEE or A2.
> > Solution becomes negative. Without this option, all ARKIMEX types work
> very
> > well and very fast.
>
> I'm glad they work as IMEX methods (please check that the solutions are
> correct), but I would like to find out why they are not working in fully
> implicit mode.  Is your problem similar to ex25?


Yes. The only difference is that I put everything (diffusion and reaction
terms) under IFunction and IJacobian.

My system is

u_t - alpha.u_xx + k.u.v = 0
v_t - beta.v_xx + k.u.v = 0

with Dirichlet boundary conditions
u(x=0,t) = u(x=L,t) = cste
v(x=0,t) = v(x=L,t) = cste

 (If I remember
> correctly, that example uses a DAE formulation for boundary conditions
> and thus has trouble with some methods with explicit stages.  There may
> be more going on.)
>

Now you say it, I'm having some troubles. After checking that it worked
very well with only diffusion terms with ARKIMEX, I introduced the reaction
term. With ARKIMEX 1BEE/A2, it goes very fast up to t=0.27 s, then, steps
start being rejected and then it goes very slowly, with small timesteps.
See output below.
Do you think it is due to the fact that I use IFunction to define boundary
conditions, which produces troubles with ARKIMEX methods ?
What would be the solution ?

Christophe

TSAdapt 'basic': step 156 accepted t=0.112524   + 2.250e-02 wlte=3.07e-05
family='arkimex' scheme=0:'1bee' dt=2.701e-02
      TSAdapt 'basic': step 157 accepted t=0.135029   + 2.701e-02
wlte=5.03e-05 family='arkimex' scheme=0:'1bee' dt=3.241e-02
      TSAdapt 'basic': step 158 accepted t=0.162034   + 3.241e-02
wlte=8.36e-05 family='arkimex' scheme=0:'1bee' dt=3.889e-02
      TSAdapt 'basic': step 159 accepted t=0.194441   + 3.889e-02
wlte=0.000141 family='arkimex' scheme=0:'1bee' dt=4.667e-02
      TSAdapt 'basic': step 160 accepted t=0.233329   + 4.667e-02
wlte=0.000244 family='arkimex' scheme=0:'1bee' dt=5.000e-02
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 5.000e-02
retrying with dt=1.250e-02
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 1.250e-02
retrying with dt=3.125e-03
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 3.125e-03
retrying with dt=7.813e-04
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 7.813e-04
retrying with dt=1.953e-04
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 1.953e-04
retrying with dt=4.883e-05
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 4.883e-05
retrying with dt=1.221e-05
      TSAdapt 'basic': step 161 stage rejected t=0.279995   + 1.221e-05
retrying with dt=3.052e-06
      TSAdapt 'basic': step 161 accepted t=0.279995   + 3.052e-06
wlte=2.58e-06 family='arkimex' scheme=0:'1bee' dt=3.662e-06
      TSAdapt 'basic': step 162 accepted t=0.279998   + 3.662e-06
wlte=3.72e-06 family='arkimex' scheme=0:'1bee' dt=4.395e-06
      TSAdapt 'basic': step 163 stage rejected t=0.280002   + 4.395e-06
retrying with dt=1.099e-06
      TSAdapt 'basic': step 163 accepted t=0.280002   + 1.099e-06
wlte=3.35e-07 family='arkimex' scheme=0:'1bee' dt=1.318e-06
      TSAdapt 'basic': step 164 accepted t=0.280003   + 1.318e-06
wlte=4.82e-07 family='arkimex' scheme=0:'1bee' dt=1.582e-06
      TSAdapt 'basic': step 165 accepted t=0.280004   + 1.582e-06
wlte=6.94e-07 family='arkimex' scheme=0:'1bee' dt=1.898e-06
      TSAdapt 'basic': step 166 accepted t=0.280006   + 1.898e-06
wlte=1e-06 family='arkimex' scheme=0:'1bee' dt=2.278e-06
      TSAdapt 'basic': step 167 accepted t=0.280008   + 2.278e-06
wlte=1.44e-06 family='arkimex' scheme=0:'1bee' dt=2.734e-06
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/6eb0b824/attachment-0001.html>

From bisheshkh at gmail.com  Wed Oct 16 09:18:01 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Wed, 16 Oct 2013 16:18:01 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
Message-ID: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>

Dear all,
I'm trying to solve a stokes flow with constant viscosity but with non-zero
divergence prescribed in the rhs.

I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p
respectively.
I have another DMDA (mDaP) of same size but of 1 dof corresponding to only
p.
I have assigned the null space for constant pressure inside the code. I
have assigned two nullspace basis:  One corresponding to vector created
from mDa that is assigned to outer ksp. Second corresponding to vector
created from mDaP that is assigned to a ksp obtained from the fieldsplit
corresponding to the schur complement.

Now when running the code, the solver converges for up to certain size,
e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is
given at the end of the emal.
But when I double the size of the grid in each dimension, it gives me a
run-time error.

The options  I've used are of the kind:
-pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0
-pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
-fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
-fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view

Here are:
1. Error message when using hypre for fieldsplit_0
2. Error message when using gamg for fieldsplit_0
3. -ksp_view of the working case using hypre for filedsplit_0

I get following error when I use hypre :
1.
******************************************************************************************************
[5]PETSC ERROR: --------------------- Error Message
------------------------------------
[5]PETSC ERROR: Signal received!
[5]PETSC ERROR:
------------------------------------------------------------------------
[5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[5]PETSC ERROR: See docs/changes/index.html for recent updates.
[5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[5]PETSC ERROR: See docs/index.html for manual pages.
[5]PETSC ERROR:
------------------------------------------------------------------------
[5]PETSC ERROR:
/epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
[5]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
[5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
[5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/
--with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[5]PETSC ERROR:
------------------------------------------------------------------------
[5]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
batch system) has told this process to end
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[6]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[6]PETSC ERROR: likely location of problem given in stack below
[6]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[6]PETSC ERROR:       INSTEAD the line number of the start of the function
[6]PETSC ERROR:       is given.
[6]PETSC ERROR: [6] HYPRE_SetupXXX line 130
/tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
[6]PETSC ERROR: [6] PCSetUp_HYPRE line 94
/tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
[6]PETSC ERROR: [6] PCSetUp line 868
/tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: [6] KSPSetUp line 192
/tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: [6] KSPSolve line 356
/tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: [6] MatMult_SchurComplement line 75
/tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
[6]PETSC ERROR: [6] MatNullSpaceTest line 408
/tmp/petsc-3.4.3/src/mat/interface/matnull.c
[6]PETSC ERROR: [6] solveModel line 113
"unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx


2.
****************************************************************************************************
Using gamg instead has errors like following:

[5]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR:       INSTEAD the line number of the start of the function
[5]PETSC ERROR:       is given.
[5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202
/tmp/petsc-3.4.3/include/petsc-private/matimpl.h
[5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124
/tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
[5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80
/tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
[5]PETSC ERROR: [5] MatPtAP line 8223
/tmp/petsc-3.4.3/src/mat/interface/matrix.c
[5]PETSC ERROR: [5] createLevel line 144
/tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
[5]PETSC ERROR: [5] PCSetUp_GAMG line 545
/tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
[5]PETSC ERROR: [5] PCSetUp line 868
/tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
[5]PETSC ERROR: [5] KSPSetUp line 192
/tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: [5] KSPSolve line 356
/tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: [5] MatMult_SchurComplement line 75
/tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
[5]PETSC ERROR: [5] MatNullSpaceTest line 408
/tmp/petsc-3.4.3/src/mat/interface/matnull.c
[5]PETSC ERROR: [5] solveModel line 113
"unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx


3.
********************************************************************************************************

BUT, It does give me results when I use a domain of size: 91X109 X 91 (half
sized in each dimension) The result along with ksp view in this case is as
follows:

Linear solve converged due to CONVERGED_RTOL iterations 2
KSP Object: 64 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
  left preconditioning
  has attached null space
  using PRECONDITIONED norm type for convergence test
PC Object: 64 MPI processes
  type: fieldsplit
    FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL
    Preconditioner for the Schur complement formed from user provided matrix
    Split info:
    Split number 0 Fields  0, 1, 2
    Split number 1 Fields  3
    KSP solver for A00 block
      KSP Object:      (fieldsplit_0_)       64 MPI processes
        type: gmres
          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
          GMRES: happy breakdown tolerance 1e-30
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using PRECONDITIONED norm type for convergence test
      PC Object:      (fieldsplit_0_)       64 MPI processes
        type: hypre
          HYPRE BoomerAMG preconditioning
          HYPRE BoomerAMG: Cycle type V
          HYPRE BoomerAMG: Maximum number of levels 25
          HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
          HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
          HYPRE BoomerAMG: Threshold for strong coupling 0.25
          HYPRE BoomerAMG: Interpolation truncation factor 0
          HYPRE BoomerAMG: Interpolation: max elements per row 0
          HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
          HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
          HYPRE BoomerAMG: Maximum row sums 0.9
          HYPRE BoomerAMG: Sweeps down         1
          HYPRE BoomerAMG: Sweeps up           1
          HYPRE BoomerAMG: Sweeps on coarse    1
          HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
          HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
          HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
          HYPRE BoomerAMG: Relax weight  (all)      1
          HYPRE BoomerAMG: Outer relax weight (all) 1
          HYPRE BoomerAMG: Using CF-relaxation
          HYPRE BoomerAMG: Measure type        local
          HYPRE BoomerAMG: Coarsen type        Falgout
          HYPRE BoomerAMG: Interpolation type  classical
        linear system matrix = precond matrix:
        Matrix Object:         64 MPI processes
          type: mpiaij
          rows=2793120, cols=2793120
          total: nonzeros=221624352, allocated nonzeros=221624352
          total number of mallocs used during MatSetValues calls =0
            using I-node (on process 0) routines: found 14812 nodes, limit
used is 5
    KSP solver for S = A11 - A10 inv(A00) A01
      KSP Object:      (fieldsplit_1_)       64 MPI processes
        type: gmres
          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
          GMRES: happy breakdown tolerance 1e-30
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        has attached null space
        using PRECONDITIONED norm type for convergence test
      PC Object:      (fieldsplit_1_)       64 MPI processes
        type: bjacobi
          block Jacobi: number of blocks = 64
          Local solve is same for all blocks, in the following KSP and PC
objects:
        KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
          type: preonly
          maximum iterations=10000, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
        PC Object:        (fieldsplit_1_sub_)         1 MPI processes
          type: ilu
            ILU: out-of-place factorization
            0 levels of fill
            tolerance for zero pivot 2.22045e-14
            using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
            matrix ordering: natural
            factor fill ratio given 1, needed 1
              Factored matrix follows:
                Matrix Object:                 1 MPI processes
                  type: seqaij
                  rows=14812, cols=14812
                  package used to perform factorization: petsc
                  total: nonzeros=368098, allocated nonzeros=368098
                  total number of mallocs used during MatSetValues calls =0
                    not using I-node routines
          linear system matrix = precond matrix:
          Matrix Object:           1 MPI processes
            type: seqaij
            rows=14812, cols=14812
            total: nonzeros=368098, allocated nonzeros=368098
            total number of mallocs used during MatSetValues calls =0
              not using I-node routines

        linear system matrix followed by preconditioner matrix:
        Matrix Object:         64 MPI processes
          type: schurcomplement
          rows=931040, cols=931040
            Schur complement A11 - A10 inv(A00) A01
            A11
              Matrix Object:               64 MPI processes
                type: mpiaij
                rows=931040, cols=931040
                total: nonzeros=24624928, allocated nonzeros=24624928
                total number of mallocs used during MatSetValues calls =0
                  not using I-node (on process 0) routines
            A10
              Matrix Object:               64 MPI processes
                type: mpiaij
                rows=931040, cols=2793120
                total: nonzeros=73874784, allocated nonzeros=73874784
                total number of mallocs used during MatSetValues calls =0
                  not using I-node (on process 0) routines
            KSP of A00
              KSP Object:              (fieldsplit_0_)               64 MPI
processes
                type: gmres
                  GMRES: restart=30, using Classical (unmodified)
Gram-Schmidt Orthogonalization with no iterative refinement
                  GMRES: happy breakdown tolerance 1e-30
                maximum iterations=10000, initial guess is zero
                tolerances:  relative=1e-05, absolute=1e-50,
divergence=10000
                left preconditioning
                using PRECONDITIONED norm type for convergence test
              PC Object:              (fieldsplit_0_)               64 MPI
processes
                type: hypre
                  HYPRE BoomerAMG preconditioning
                  HYPRE BoomerAMG: Cycle type V
                  HYPRE BoomerAMG: Maximum number of levels 25
                  HYPRE BoomerAMG: Maximum number of iterations PER hypre
call 1
                  HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
                  HYPRE BoomerAMG: Threshold for strong coupling 0.25
                  HYPRE BoomerAMG: Interpolation truncation factor 0
                  HYPRE BoomerAMG: Interpolation: max elements per row 0
                  HYPRE BoomerAMG: Number of levels of aggressive
coarsening 0
                  HYPRE BoomerAMG: Number of paths for aggressive
coarsening 1
                  HYPRE BoomerAMG: Maximum row sums 0.9
                  HYPRE BoomerAMG: Sweeps down         1
                  HYPRE BoomerAMG: Sweeps up           1
                  HYPRE BoomerAMG: Sweeps on coarse    1
                  HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
                  HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
                  HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
                  HYPRE BoomerAMG: Relax weight  (all)      1
                  HYPRE BoomerAMG: Outer relax weight (all) 1
                  HYPRE BoomerAMG: Using CF-relaxation
                  HYPRE BoomerAMG: Measure type        local
                  HYPRE BoomerAMG: Coarsen type        Falgout
                  HYPRE BoomerAMG: Interpolation type  classical
                linear system matrix = precond matrix:
                Matrix Object:                 64 MPI processes
                  type: mpiaij
                  rows=2793120, cols=2793120
                  total: nonzeros=221624352, allocated nonzeros=221624352
                  total number of mallocs used during MatSetValues calls =0
                    using I-node (on process 0) routines: found 14812
nodes, limit used is 5
            A01
              Matrix Object:               64 MPI processes
                type: mpiaij
                rows=2793120, cols=931040
                total: nonzeros=73874784, allocated nonzeros=73874784
                total number of mallocs used during MatSetValues calls =0
                  using I-node (on process 0) routines: found 14812 nodes,
limit used is 5
        Matrix Object:         64 MPI processes
          type: mpiaij
          rows=931040, cols=931040
          total: nonzeros=24624928, allocated nonzeros=24624928
          total number of mallocs used during MatSetValues calls =0
            not using I-node (on process 0) routines
  linear system matrix = precond matrix:
  Matrix Object:   64 MPI processes
    type: mpiaij
    rows=3724160, cols=3724160, bs=4
    total: nonzeros=393998848, allocated nonzeros=393998848
    total number of mallocs used during MatSetValues calls =0

******************************************************************************************************
What could be going wrong here ? Is it something related to null-space
setting ? But I do not know why it does not arise for smaller domain sizes!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/b95f3755/attachment-0001.html>

From dave.mayhem23 at gmail.com  Wed Oct 16 09:32:56 2013
From: dave.mayhem23 at gmail.com (Dave May)
Date: Wed, 16 Oct 2013 16:32:56 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
Message-ID: <CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>

Sounds like a memory error.
I'd run your code through valgrind to double check. The error could be
completely unconnected to the nullspaces.

Cheers,
  Dave


On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> Dear all,
> I'm trying to solve a stokes flow with constant viscosity but with
> non-zero divergence prescribed in the rhs.
>
> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p
> respectively.
> I have another DMDA (mDaP) of same size but of 1 dof corresponding to only
> p.
> I have assigned the null space for constant pressure inside the code. I
> have assigned two nullspace basis:  One corresponding to vector created
> from mDa that is assigned to outer ksp. Second corresponding to vector
> created from mDaP that is assigned to a ksp obtained from the fieldsplit
> corresponding to the schur complement.
>
> Now when running the code, the solver converges for up to certain size,
> e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is
> given at the end of the emal.
> But when I double the size of the grid in each dimension, it gives me a
> run-time error.
>
> The options  I've used are of the kind:
> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0
> -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
> -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
>
> Here are:
> 1. Error message when using hypre for fieldsplit_0
> 2. Error message when using gamg for fieldsplit_0
> 3. -ksp_view of the working case using hypre for filedsplit_0
>
> I get following error when I use hypre :
> 1.
> ******************************************************************************************************
> [5]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [5]PETSC ERROR: Signal received!
> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [5]PETSC ERROR: See docs/index.html for manual pages.
> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> [5]PETSC ERROR:
> /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
> arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
> [5]PETSC ERROR: Libraries linked from
> /epi/asclepios2/bkhanal/petscDebug/lib
> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
> [5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/
> --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> [5]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [6]PETSC ERROR:
> ------------------------------------------------------------------------
> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
> batch system) has told this process to end
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [6]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [6]PETSC ERROR: likely location of problem given in stack below
> [6]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [6]PETSC ERROR:       INSTEAD the line number of the start of the function
> [6]PETSC ERROR:       is given.
> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130
> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94
> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> [6]PETSC ERROR: [6] PCSetUp line 868
> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> [6]PETSC ERROR: [6] KSPSetUp line 192
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: [6] KSPSolve line 356
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75
> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> [6]PETSC ERROR: [6] MatNullSpaceTest line 408
> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> [6]PETSC ERROR: [6] solveModel line 113
> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>
>
> 2.
> ****************************************************************************************************
> Using gamg instead has errors like following:
>
> [5]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [5]PETSC ERROR:       INSTEAD the line number of the start of the function
> [5]PETSC ERROR:       is given.
> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202
> /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124
> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80
> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> [5]PETSC ERROR: [5] MatPtAP line 8223
> /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> [5]PETSC ERROR: [5] createLevel line 144
> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545
> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> [5]PETSC ERROR: [5] PCSetUp line 868
> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> [5]PETSC ERROR: [5] KSPSetUp line 192
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [5]PETSC ERROR: [5] KSPSolve line 356
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75
> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> [5]PETSC ERROR: [5] MatNullSpaceTest line 408
> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> [5]PETSC ERROR: [5] solveModel line 113
> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>
>
> 3.
> ********************************************************************************************************
>
> BUT, It does give me results when I use a domain of size: 91X109 X 91
> (half sized in each dimension) The result along with ksp view in this case
> is as follows:
>
> Linear solve converged due to CONVERGED_RTOL iterations 2
> KSP Object: 64 MPI processes
>   type: gmres
>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>   left preconditioning
>   has attached null space
>   using PRECONDITIONED norm type for convergence test
> PC Object: 64 MPI processes
>   type: fieldsplit
>     FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL
>     Preconditioner for the Schur complement formed from user provided
> matrix
>     Split info:
>     Split number 0 Fields  0, 1, 2
>     Split number 1 Fields  3
>     KSP solver for A00 block
>       KSP Object:      (fieldsplit_0_)       64 MPI processes
>         type: gmres
>           GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>           GMRES: happy breakdown tolerance 1e-30
>         maximum iterations=10000, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>         left preconditioning
>         using PRECONDITIONED norm type for convergence test
>       PC Object:      (fieldsplit_0_)       64 MPI processes
>         type: hypre
>           HYPRE BoomerAMG preconditioning
>           HYPRE BoomerAMG: Cycle type V
>           HYPRE BoomerAMG: Maximum number of levels 25
>           HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>           HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>           HYPRE BoomerAMG: Threshold for strong coupling 0.25
>           HYPRE BoomerAMG: Interpolation truncation factor 0
>           HYPRE BoomerAMG: Interpolation: max elements per row 0
>           HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>           HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>           HYPRE BoomerAMG: Maximum row sums 0.9
>           HYPRE BoomerAMG: Sweeps down         1
>           HYPRE BoomerAMG: Sweeps up           1
>           HYPRE BoomerAMG: Sweeps on coarse    1
>           HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>           HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>           HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>           HYPRE BoomerAMG: Relax weight  (all)      1
>           HYPRE BoomerAMG: Outer relax weight (all) 1
>           HYPRE BoomerAMG: Using CF-relaxation
>           HYPRE BoomerAMG: Measure type        local
>           HYPRE BoomerAMG: Coarsen type        Falgout
>           HYPRE BoomerAMG: Interpolation type  classical
>         linear system matrix = precond matrix:
>         Matrix Object:         64 MPI processes
>           type: mpiaij
>           rows=2793120, cols=2793120
>           total: nonzeros=221624352, allocated nonzeros=221624352
>           total number of mallocs used during MatSetValues calls =0
>             using I-node (on process 0) routines: found 14812 nodes, limit
> used is 5
>     KSP solver for S = A11 - A10 inv(A00) A01
>       KSP Object:      (fieldsplit_1_)       64 MPI processes
>         type: gmres
>           GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>           GMRES: happy breakdown tolerance 1e-30
>         maximum iterations=10000, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>         left preconditioning
>         has attached null space
>         using PRECONDITIONED norm type for convergence test
>       PC Object:      (fieldsplit_1_)       64 MPI processes
>         type: bjacobi
>           block Jacobi: number of blocks = 64
>           Local solve is same for all blocks, in the following KSP and PC
> objects:
>         KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
>           type: preonly
>           maximum iterations=10000, initial guess is zero
>           tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>           left preconditioning
>           using NONE norm type for convergence test
>         PC Object:        (fieldsplit_1_sub_)         1 MPI processes
>           type: ilu
>             ILU: out-of-place factorization
>             0 levels of fill
>             tolerance for zero pivot 2.22045e-14
>             using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>             matrix ordering: natural
>             factor fill ratio given 1, needed 1
>               Factored matrix follows:
>                 Matrix Object:                 1 MPI processes
>                   type: seqaij
>                   rows=14812, cols=14812
>                   package used to perform factorization: petsc
>                   total: nonzeros=368098, allocated nonzeros=368098
>                   total number of mallocs used during MatSetValues calls =0
>                     not using I-node routines
>           linear system matrix = precond matrix:
>           Matrix Object:           1 MPI processes
>             type: seqaij
>             rows=14812, cols=14812
>             total: nonzeros=368098, allocated nonzeros=368098
>             total number of mallocs used during MatSetValues calls =0
>               not using I-node routines
>
>         linear system matrix followed by preconditioner matrix:
>         Matrix Object:         64 MPI processes
>           type: schurcomplement
>           rows=931040, cols=931040
>             Schur complement A11 - A10 inv(A00) A01
>             A11
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=931040, cols=931040
>                 total: nonzeros=24624928, allocated nonzeros=24624928
>                 total number of mallocs used during MatSetValues calls =0
>                   not using I-node (on process 0) routines
>             A10
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=931040, cols=2793120
>                 total: nonzeros=73874784, allocated nonzeros=73874784
>                 total number of mallocs used during MatSetValues calls =0
>                   not using I-node (on process 0) routines
>             KSP of A00
>               KSP Object:              (fieldsplit_0_)               64
> MPI processes
>                 type: gmres
>                   GMRES: restart=30, using Classical (unmodified)
> Gram-Schmidt Orthogonalization with no iterative refinement
>                   GMRES: happy breakdown tolerance 1e-30
>                 maximum iterations=10000, initial guess is zero
>                 tolerances:  relative=1e-05, absolute=1e-50,
> divergence=10000
>                 left preconditioning
>                 using PRECONDITIONED norm type for convergence test
>               PC Object:              (fieldsplit_0_)               64 MPI
> processes
>                 type: hypre
>                   HYPRE BoomerAMG preconditioning
>                   HYPRE BoomerAMG: Cycle type V
>                   HYPRE BoomerAMG: Maximum number of levels 25
>                   HYPRE BoomerAMG: Maximum number of iterations PER hypre
> call 1
>                   HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>                   HYPRE BoomerAMG: Threshold for strong coupling 0.25
>                   HYPRE BoomerAMG: Interpolation truncation factor 0
>                   HYPRE BoomerAMG: Interpolation: max elements per row 0
>                   HYPRE BoomerAMG: Number of levels of aggressive
> coarsening 0
>                   HYPRE BoomerAMG: Number of paths for aggressive
> coarsening 1
>                   HYPRE BoomerAMG: Maximum row sums 0.9
>                   HYPRE BoomerAMG: Sweeps down         1
>                   HYPRE BoomerAMG: Sweeps up           1
>                   HYPRE BoomerAMG: Sweeps on coarse    1
>                   HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>                   HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>                   HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>                   HYPRE BoomerAMG: Relax weight  (all)      1
>                   HYPRE BoomerAMG: Outer relax weight (all) 1
>                   HYPRE BoomerAMG: Using CF-relaxation
>                   HYPRE BoomerAMG: Measure type        local
>                   HYPRE BoomerAMG: Coarsen type        Falgout
>                   HYPRE BoomerAMG: Interpolation type  classical
>                 linear system matrix = precond matrix:
>                 Matrix Object:                 64 MPI processes
>                   type: mpiaij
>                   rows=2793120, cols=2793120
>                   total: nonzeros=221624352, allocated nonzeros=221624352
>                   total number of mallocs used during MatSetValues calls =0
>                     using I-node (on process 0) routines: found 14812
> nodes, limit used is 5
>             A01
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=2793120, cols=931040
>                 total: nonzeros=73874784, allocated nonzeros=73874784
>                 total number of mallocs used during MatSetValues calls =0
>                   using I-node (on process 0) routines: found 14812 nodes,
> limit used is 5
>         Matrix Object:         64 MPI processes
>           type: mpiaij
>           rows=931040, cols=931040
>           total: nonzeros=24624928, allocated nonzeros=24624928
>           total number of mallocs used during MatSetValues calls =0
>             not using I-node (on process 0) routines
>   linear system matrix = precond matrix:
>   Matrix Object:   64 MPI processes
>     type: mpiaij
>     rows=3724160, cols=3724160, bs=4
>     total: nonzeros=393998848, allocated nonzeros=393998848
>     total number of mallocs used during MatSetValues calls =0
>
>
> ******************************************************************************************************
> What could be going wrong here ? Is it something related to null-space
> setting ? But I do not know why it does not arise for smaller domain sizes!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/7019c46a/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed Oct 16 09:36:41 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 09:36:41 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector
	for a 3-D domain
In-Reply-To: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>
References: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>
Message-ID: <87fvs1mgee.fsf@mcs.anl.gov>

Roc Wang <pengxwang at hotmail.com> writes:
>     I have a DM and its vector for a structured 3-D domain and trying
>     to extract the data for a 2-D plane.  I did some research on the
>     manual and noticed there is function named DMSlicedCreate(),
>     however, it is for unstructured problem. Is there any ohter
>     function to extract vector for a 2-D plane from DM vector for a
>     3-D domain directly?

Create a VecScatter.  Do you want to extract many planes to separate
communicators?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/cb745264/attachment.pgp>

From berndt at lanl.gov  Wed Oct 16 09:43:33 2013
From: berndt at lanl.gov (Markus Berndt)
Date: Wed, 16 Oct 2013 08:43:33 -0600
Subject: [petsc-users] Can I eliminate all mallocs in parallel matrix
	assembly?
Message-ID: <525EA615.40006@lanl.gov>

I am trying to minimize the memory footprint of an application by 
specifying exactly how many nonzeros are needed per row of the matrix. I 
am pretty sure that I am correctly calculating these memory requirements 
for the diagonal and off diagonal parts of the parallel matrix (I am 
using MatCreateAIJ).

When running the application with the -info option I see a fairly small 
number of mallocs:

[0] MatStashScatterBegin_Private(): No of messages: 2
[0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 13192
[0] MatStashScatterBegin_Private(): Mesg_to: 2: size: 14280
[0] MatAssemblyBegin_MPIAIJ(): Stash has 3432 entries, uses 0 mallocs.
[1] MatAssemblyBegin_MPIAIJ(): Stash has 2912 entries, uses 0 mallocs.
[2] MatAssemblyBegin_MPIAIJ(): Stash has 2616 entries, uses 0 mallocs.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 692 X 692; storage space: 337 
unneeded,13896 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 25
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 718 X 718; storage space: 318 
unneeded,14830 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 24
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27
[2] MatAssemblyEnd_SeqAIJ(): Matrix size: 787 X 787; storage space: 441 
unneeded,15763 used
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 33
[2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 27
[2] Mat_CheckInode(): Found 787 nodes out of 787 rows. Not using Inode 
routines
[0] Mat_CheckInode(): Found 692 nodes out of 692 rows. Not using Inode 
routines
[1] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
64966848
[2] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
66809232
[0] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
53202336
[2] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
66809232
[1] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
64966848
[0] PetscCommDuplicate(): Using internal PETSc communicator 34964256 
53202336
[0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
[0] VecScatterCreate(): General case: MPI to Seq
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 692 X 363; storage space: 370 
unneeded,2091 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 21
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 718 X 334; storage space: 344 
unneeded,1980 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 21
[2] MatAssemblyEnd_SeqAIJ(): Matrix size: 787 X 301; storage space: 362 
unneeded,2093 used
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 20

This example runs on three PEs.

I am wondering if I should be able to eliminate all mallocs? Instead 
should my goal be to eliminate not all but most mallocs with a small 
number of mallocs during matrix assembly being acceptable?

Thanks

- Markus


-- 
Markus Berndt - CCS-2 - LANL - 505-665-4711


From pengxwang at hotmail.com  Wed Oct 16 09:48:16 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Wed, 16 Oct 2013 09:48:16 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector
 for a 3-D domain
In-Reply-To: <87fvs1mgee.fsf@mcs.anl.gov>
References: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>,
	<87fvs1mgee.fsf@mcs.anl.gov>
Message-ID: <BAY179-W6247DB7F23AE22470365D6B4040@phx.gbl>



> From: jedbrown at mcs.anl.gov
> To: pengxwang at hotmail.com; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] extract vector for a 2-D plane from a DM vector for a 3-D domain
> Date: Wed, 16 Oct 2013 09:36:41 -0500
> 
> Roc Wang <pengxwang at hotmail.com> writes:
> >     I have a DM and its vector for a structured 3-D domain and trying
> >     to extract the data for a 2-D plane.  I did some research on the
> >     manual and noticed there is function named DMSlicedCreate(),
> >     however, it is for unstructured problem. Is there any ohter
> >     function to extract vector for a 2-D plane from DM vector for a
> >     3-D domain directly?
> 
> Create a VecScatter.  Do you want to extract many planes to separate
> communicators?
Is it possible I have both cases? 1 a few planes in the same communicator;  2 many planes to separate communicator.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/0fcea17c/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 16 09:57:39 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 09:57:39 -0500
Subject: [petsc-users] Can I eliminate all mallocs in parallel matrix
	assembly?
In-Reply-To: <525EA615.40006@lanl.gov>
References: <525EA615.40006@lanl.gov>
Message-ID: <874n8hmffg.fsf@mcs.anl.gov>

Markus Berndt <berndt at lanl.gov> writes:

> I am trying to minimize the memory footprint of an application by 
> specifying exactly how many nonzeros are needed per row of the matrix. I 
> am pretty sure that I am correctly calculating these memory requirements 
> for the diagonal and off diagonal parts of the parallel matrix (I am 
> using MatCreateAIJ).

Either you have called another function later that clears the
preallocation information or the information you provided is incorrect.

What version of PETSc are you using?  We currently set
MAT_NEW_NONZERO_ALLOCATION_ERR by default so you should get an error
when a new nonzero triggers allocation.  You can catch that in a
debugger and figure out why there are more nonzeros than you allocated
for.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/c08b0a70/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 16 09:59:34 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 09:59:34 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector
	for a 3-D domain
In-Reply-To: <BAY179-W6247DB7F23AE22470365D6B4040@phx.gbl>
References: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>
	<87fvs1mgee.fsf@mcs.anl.gov>
	<BAY179-W6247DB7F23AE22470365D6B4040@phx.gbl>
Message-ID: <871u3lmfc9.fsf@mcs.anl.gov>

Roc Wang <pengxwang at hotmail.com> writes:
> Is it possible I have both cases? 1 a few planes in the same
> communicator; 2 many planes to separate communicator.

Yeah, just create the VecScatter however you like.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/306f8520/attachment.pgp>

From berndt at lanl.gov  Wed Oct 16 10:10:40 2013
From: berndt at lanl.gov (Markus Berndt)
Date: Wed, 16 Oct 2013 09:10:40 -0600
Subject: [petsc-users] Can I eliminate all mallocs in parallel matrix
 assembly?
In-Reply-To: <874n8hmffg.fsf@mcs.anl.gov>
References: <525EA615.40006@lanl.gov> <874n8hmffg.fsf@mcs.anl.gov>
Message-ID: <525EAC70.7050102@lanl.gov>

I am setting MAT_NEW_NONZERO_ALLOCATION_ERR to PETSC_FALSE, the PETSc 
version is 3.3-p6.

Thanks for clarifying this. Just to make sure I understand correctly, by 
providing the correct values in d_nnz and o_nnz I should always be able 
to avoid any mallocs in matrix assembly?

I will dig in with a debugger as you suggest

Thanks

- Markus

On 10/16/2013 08:57 AM, Jed Brown wrote:
> Markus Berndt <berndt at lanl.gov> writes:
>
>> I am trying to minimize the memory footprint of an application by
>> specifying exactly how many nonzeros are needed per row of the matrix. I
>> am pretty sure that I am correctly calculating these memory requirements
>> for the diagonal and off diagonal parts of the parallel matrix (I am
>> using MatCreateAIJ).
> Either you have called another function later that clears the
> preallocation information or the information you provided is incorrect.
>
> What version of PETSc are you using?  We currently set
> MAT_NEW_NONZERO_ALLOCATION_ERR by default so you should get an error
> when a new nonzero triggers allocation.  You can catch that in a
> debugger and figure out why there are more nonzeros than you allocated
> for.

-- 
Markus Berndt - CCS-2 - LANL - 505-665-4711


From knepley at gmail.com  Wed Oct 16 10:18:01 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Oct 2013 10:18:01 -0500
Subject: [petsc-users] Can I eliminate all mallocs in parallel matrix
	assembly?
In-Reply-To: <525EAC70.7050102@lanl.gov>
References: <525EA615.40006@lanl.gov> <874n8hmffg.fsf@mcs.anl.gov>
	<525EAC70.7050102@lanl.gov>
Message-ID: <CAMYG4Gm3aF=Yk360tUCCPAht1eXZS8e3qHgxHyLCiEbqjx-fQQ@mail.gmail.com>

On Wed, Oct 16, 2013 at 10:10 AM, Markus Berndt <berndt at lanl.gov> wrote:

> I am setting MAT_NEW_NONZERO_ALLOCATION_ERR to PETSC_FALSE, the PETSc
> version is 3.3-p6.
>
> Thanks for clarifying this. Just to make sure I understand correctly, by
> providing the correct values in d_nnz and o_nnz I should always be able to
> avoid any mallocs in matrix assembly?
>

Yes.

   Matt


> I will dig in with a debugger as you suggest
>
> Thanks
>
> - Markus
>
>
> On 10/16/2013 08:57 AM, Jed Brown wrote:
>
>> Markus Berndt <berndt at lanl.gov> writes:
>>
>>  I am trying to minimize the memory footprint of an application by
>>> specifying exactly how many nonzeros are needed per row of the matrix. I
>>> am pretty sure that I am correctly calculating these memory requirements
>>> for the diagonal and off diagonal parts of the parallel matrix (I am
>>> using MatCreateAIJ).
>>>
>> Either you have called another function later that clears the
>> preallocation information or the information you provided is incorrect.
>>
>> What version of PETSc are you using?  We currently set
>> MAT_NEW_NONZERO_ALLOCATION_ERR by default so you should get an error
>> when a new nonzero triggers allocation.  You can catch that in a
>> debugger and figure out why there are more nonzeros than you allocated
>> for.
>>
>
> --
> Markus Berndt - CCS-2 - LANL - 505-665-4711
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/dbe9231f/attachment-0001.html>

From bsmith at mcs.anl.gov  Wed Oct 16 10:19:04 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 16 Oct 2013 10:19:04 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
Message-ID: <4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>


http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

especially the GAMG version which you also try in the debugger with the argument -start_in_debugger



On Oct 16, 2013, at 9:32 AM, Dave May <dave.mayhem23 at gmail.com> wrote:

> Sounds like a memory error.
> I'd run your code through valgrind to double check. The error could be completely unconnected to the nullspaces.
> 
> Cheers,
>   Dave 
> 
> 
> On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:
> Dear all,
> I'm trying to solve a stokes flow with constant viscosity but with non-zero divergence prescribed in the rhs.
> 
> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p respectively.
> I have another DMDA (mDaP) of same size but of 1 dof corresponding to only p.
> I have assigned the null space for constant pressure inside the code. I have assigned two nullspace basis:  One corresponding to vector created from mDa that is assigned to outer ksp. Second corresponding to vector created from mDaP that is assigned to a ksp obtained from the fieldsplit corresponding to the schur complement. 
> 
> Now when running the code, the solver converges for up to certain size, e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is given at the end of the emal.
> But when I double the size of the grid in each dimension, it gives me a run-time error.
> 
> The options  I've used are of the kind:  
> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
> 
> Here are: 
> 1. Error message when using hypre for fieldsplit_0
> 2. Error message when using gamg for fieldsplit_0
> 3. -ksp_view of the working case using hypre for filedsplit_0
> 
> I get following error when I use hypre :
> 1. ******************************************************************************************************
> [5]PETSC ERROR: --------------------- Error Message ------------------------------------
> [5]PETSC ERROR: Signal received!
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [5]PETSC ERROR: See docs/index.html for manual pages.
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
> [5]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
> [5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1 --download-metis --download-parmetis --download-superlu_dist --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [6]PETSC ERROR: likely location of problem given in stack below
> [6]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [6]PETSC ERROR:       INSTEAD the line number of the start of the function
> [6]PETSC ERROR:       is given.
> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> [6]PETSC ERROR: [6] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> [6]PETSC ERROR: [6] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: [6] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> [6]PETSC ERROR: [6] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> [6]PETSC ERROR: [6] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> 
> 
> 2. ****************************************************************************************************
> Using gamg instead has errors like following:
> 
> [5]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [5]PETSC ERROR:       INSTEAD the line number of the start of the function
> [5]PETSC ERROR:       is given.
> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202 /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> [5]PETSC ERROR: [5] MatPtAP line 8223 /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> [5]PETSC ERROR: [5] createLevel line 144 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> [5]PETSC ERROR: [5] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> [5]PETSC ERROR: [5] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [5]PETSC ERROR: [5] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> [5]PETSC ERROR: [5] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> [5]PETSC ERROR: [5] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> 
> 
> 3. ********************************************************************************************************
> 
> BUT, It does give me results when I use a domain of size: 91X109 X 91 (half sized in each dimension) The result along with ksp view in this case is as follows:
> 
> Linear solve converged due to CONVERGED_RTOL iterations 2
> KSP Object: 64 MPI processes
>   type: gmres
>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>   left preconditioning
>   has attached null space
>   using PRECONDITIONED norm type for convergence test
> PC Object: 64 MPI processes
>   type: fieldsplit
>     FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL
>     Preconditioner for the Schur complement formed from user provided matrix
>     Split info:
>     Split number 0 Fields  0, 1, 2
>     Split number 1 Fields  3
>     KSP solver for A00 block 
>       KSP Object:      (fieldsplit_0_)       64 MPI processes
>         type: gmres
>           GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>           GMRES: happy breakdown tolerance 1e-30
>         maximum iterations=10000, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>         left preconditioning
>         using PRECONDITIONED norm type for convergence test
>       PC Object:      (fieldsplit_0_)       64 MPI processes
>         type: hypre
>           HYPRE BoomerAMG preconditioning
>           HYPRE BoomerAMG: Cycle type V
>           HYPRE BoomerAMG: Maximum number of levels 25
>           HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>           HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>           HYPRE BoomerAMG: Threshold for strong coupling 0.25
>           HYPRE BoomerAMG: Interpolation truncation factor 0
>           HYPRE BoomerAMG: Interpolation: max elements per row 0
>           HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>           HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>           HYPRE BoomerAMG: Maximum row sums 0.9
>           HYPRE BoomerAMG: Sweeps down         1
>           HYPRE BoomerAMG: Sweeps up           1
>           HYPRE BoomerAMG: Sweeps on coarse    1
>           HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>           HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>           HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>           HYPRE BoomerAMG: Relax weight  (all)      1
>           HYPRE BoomerAMG: Outer relax weight (all) 1
>           HYPRE BoomerAMG: Using CF-relaxation
>           HYPRE BoomerAMG: Measure type        local
>           HYPRE BoomerAMG: Coarsen type        Falgout
>           HYPRE BoomerAMG: Interpolation type  classical
>         linear system matrix = precond matrix:
>         Matrix Object:         64 MPI processes
>           type: mpiaij
>           rows=2793120, cols=2793120
>           total: nonzeros=221624352, allocated nonzeros=221624352
>           total number of mallocs used during MatSetValues calls =0
>             using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>     KSP solver for S = A11 - A10 inv(A00) A01 
>       KSP Object:      (fieldsplit_1_)       64 MPI processes
>         type: gmres
>           GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>           GMRES: happy breakdown tolerance 1e-30
>         maximum iterations=10000, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>         left preconditioning
>         has attached null space
>         using PRECONDITIONED norm type for convergence test
>       PC Object:      (fieldsplit_1_)       64 MPI processes
>         type: bjacobi
>           block Jacobi: number of blocks = 64
>           Local solve is same for all blocks, in the following KSP and PC objects:
>         KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
>           type: preonly
>           maximum iterations=10000, initial guess is zero
>           tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>           left preconditioning
>           using NONE norm type for convergence test
>         PC Object:        (fieldsplit_1_sub_)         1 MPI processes
>           type: ilu
>             ILU: out-of-place factorization
>             0 levels of fill
>             tolerance for zero pivot 2.22045e-14
>             using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>             matrix ordering: natural
>             factor fill ratio given 1, needed 1
>               Factored matrix follows:
>                 Matrix Object:                 1 MPI processes
>                   type: seqaij
>                   rows=14812, cols=14812
>                   package used to perform factorization: petsc
>                   total: nonzeros=368098, allocated nonzeros=368098
>                   total number of mallocs used during MatSetValues calls =0
>                     not using I-node routines
>           linear system matrix = precond matrix:
>           Matrix Object:           1 MPI processes
>             type: seqaij
>             rows=14812, cols=14812
>             total: nonzeros=368098, allocated nonzeros=368098
>             total number of mallocs used during MatSetValues calls =0
>               not using I-node routines
> 
>         linear system matrix followed by preconditioner matrix:
>         Matrix Object:         64 MPI processes
>           type: schurcomplement
>           rows=931040, cols=931040
>             Schur complement A11 - A10 inv(A00) A01
>             A11
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=931040, cols=931040
>                 total: nonzeros=24624928, allocated nonzeros=24624928
>                 total number of mallocs used during MatSetValues calls =0
>                   not using I-node (on process 0) routines
>             A10
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=931040, cols=2793120
>                 total: nonzeros=73874784, allocated nonzeros=73874784
>                 total number of mallocs used during MatSetValues calls =0
>                   not using I-node (on process 0) routines
>             KSP of A00
>               KSP Object:              (fieldsplit_0_)               64 MPI processes
>                 type: gmres
>                   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>                   GMRES: happy breakdown tolerance 1e-30
>                 maximum iterations=10000, initial guess is zero
>                 tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>                 left preconditioning
>                 using PRECONDITIONED norm type for convergence test
>               PC Object:              (fieldsplit_0_)               64 MPI processes
>                 type: hypre
>                   HYPRE BoomerAMG preconditioning
>                   HYPRE BoomerAMG: Cycle type V
>                   HYPRE BoomerAMG: Maximum number of levels 25
>                   HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>                   HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>                   HYPRE BoomerAMG: Threshold for strong coupling 0.25
>                   HYPRE BoomerAMG: Interpolation truncation factor 0
>                   HYPRE BoomerAMG: Interpolation: max elements per row 0
>                   HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>                   HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>                   HYPRE BoomerAMG: Maximum row sums 0.9
>                   HYPRE BoomerAMG: Sweeps down         1
>                   HYPRE BoomerAMG: Sweeps up           1
>                   HYPRE BoomerAMG: Sweeps on coarse    1
>                   HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>                   HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>                   HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>                   HYPRE BoomerAMG: Relax weight  (all)      1
>                   HYPRE BoomerAMG: Outer relax weight (all) 1
>                   HYPRE BoomerAMG: Using CF-relaxation
>                   HYPRE BoomerAMG: Measure type        local
>                   HYPRE BoomerAMG: Coarsen type        Falgout
>                   HYPRE BoomerAMG: Interpolation type  classical
>                 linear system matrix = precond matrix:
>                 Matrix Object:                 64 MPI processes
>                   type: mpiaij
>                   rows=2793120, cols=2793120
>                   total: nonzeros=221624352, allocated nonzeros=221624352
>                   total number of mallocs used during MatSetValues calls =0
>                     using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>             A01
>               Matrix Object:               64 MPI processes
>                 type: mpiaij
>                 rows=2793120, cols=931040
>                 total: nonzeros=73874784, allocated nonzeros=73874784
>                 total number of mallocs used during MatSetValues calls =0
>                   using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>         Matrix Object:         64 MPI processes
>           type: mpiaij
>           rows=931040, cols=931040
>           total: nonzeros=24624928, allocated nonzeros=24624928
>           total number of mallocs used during MatSetValues calls =0
>             not using I-node (on process 0) routines
>   linear system matrix = precond matrix:
>   Matrix Object:   64 MPI processes
>     type: mpiaij
>     rows=3724160, cols=3724160, bs=4
>     total: nonzeros=393998848, allocated nonzeros=393998848
>     total number of mallocs used during MatSetValues calls =0
> 
> ******************************************************************************************************
> What could be going wrong here ? Is it something related to null-space setting ? But I do not know why it does not arise for smaller domain sizes!
> 


From pengxwang at hotmail.com  Wed Oct 16 10:47:45 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Wed, 16 Oct 2013 10:47:45 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector
 for a 3-D domain
In-Reply-To: <871u3lmfc9.fsf@mcs.anl.gov>
References: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>
	<87fvs1mgee.fsf@mcs.anl.gov>
	<BAY179-W6247DB7F23AE22470365D6B4040@phx.gbl>,
	<871u3lmfc9.fsf@mcs.anl.gov>
Message-ID: <BAY179-W1598052A1FEADA1CFAD00CB4040@phx.gbl>



> From: jedbrown at mcs.anl.gov
> To: pengxwang at hotmail.com; petsc-users at mcs.anl.gov
> Subject: RE: [petsc-users] extract vector for a 2-D plane from a DM vector for a 3-D domain
> Date: Wed, 16 Oct 2013 09:59:34 -0500
> 
> Roc Wang <pengxwang at hotmail.com> writes:
> > Is it possible I have both cases? 1 a few planes in the same
> > communicator; 2 many planes to separate communicator.
> 
> Yeah, just create the VecScatter however you like.

I studied some examples for VecScatter.  To my understand, the pathway of extracting the plane is through extracting indexes (IS idx_to) of the 2-D plane from the vector of the 3-D domain(IS idx_from). To do this, I have to create an array of index for IS idx_to and create the corresponding IS by ISCreateGeneral(). Am I right?  Thanks for your informative response.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/4bcbe9ff/attachment.html>

From mfadams at lbl.gov  Wed Oct 16 10:50:48 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Wed, 16 Oct 2013 11:50:48 -0400
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
Message-ID: <44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>

You might also test with Jacobi as a sanity check.

On Oct 16, 2013, at 11:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> 
> especially the GAMG version which you also try in the debugger with the argument -start_in_debugger
> 
> 
> 
> On Oct 16, 2013, at 9:32 AM, Dave May <dave.mayhem23 at gmail.com> wrote:
> 
>> Sounds like a memory error.
>> I'd run your code through valgrind to double check. The error could be completely unconnected to the nullspaces.
>> 
>> Cheers,
>>  Dave 
>> 
>> 
>> On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:
>> Dear all,
>> I'm trying to solve a stokes flow with constant viscosity but with non-zero divergence prescribed in the rhs.
>> 
>> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p respectively.
>> I have another DMDA (mDaP) of same size but of 1 dof corresponding to only p.
>> I have assigned the null space for constant pressure inside the code. I have assigned two nullspace basis:  One corresponding to vector created from mDa that is assigned to outer ksp. Second corresponding to vector created from mDaP that is assigned to a ksp obtained from the fieldsplit corresponding to the schur complement. 
>> 
>> Now when running the code, the solver converges for up to certain size, e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is given at the end of the emal.
>> But when I double the size of the grid in each dimension, it gives me a run-time error.
>> 
>> The options  I've used are of the kind:  
>> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
>> 
>> Here are: 
>> 1. Error message when using hypre for fieldsplit_0
>> 2. Error message when using gamg for fieldsplit_0
>> 3. -ksp_view of the working case using hypre for filedsplit_0
>> 
>> I get following error when I use hypre :
>> 1. ******************************************************************************************************
>> [5]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [5]PETSC ERROR: Signal received!
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
>> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [5]PETSC ERROR: See docs/index.html for manual pages.
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
>> [5]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
>> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
>> [5]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1 --download-metis --download-parmetis --download-superlu_dist --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
>> [5]PETSC ERROR: ------------------------------------------------------------------------
>> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
>> [6]PETSC ERROR: ------------------------------------------------------------------------
>> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
>> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [6]PETSC ERROR: likely location of problem given in stack below
>> [6]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [6]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [6]PETSC ERROR:       is given.
>> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94 /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> [6]PETSC ERROR: [6] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> [6]PETSC ERROR: [6] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [6]PETSC ERROR: [6] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> [6]PETSC ERROR: [6] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> [6]PETSC ERROR: [6] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>> 
>> 
>> 2. ****************************************************************************************************
>> Using gamg instead has errors like following:
>> 
>> [5]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [5]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [5]PETSC ERROR:       is given.
>> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202 /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
>> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> [5]PETSC ERROR: [5] MatPtAP line 8223 /tmp/petsc-3.4.3/src/mat/interface/matrix.c
>> [5]PETSC ERROR: [5] createLevel line 144 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545 /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> [5]PETSC ERROR: [5] PCSetUp line 868 /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> [5]PETSC ERROR: [5] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [5]PETSC ERROR: [5] KSPSolve line 356 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75 /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> [5]PETSC ERROR: [5] MatNullSpaceTest line 408 /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> [5]PETSC ERROR: [5] solveModel line 113 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>> 
>> 
>> 3. ********************************************************************************************************
>> 
>> BUT, It does give me results when I use a domain of size: 91X109 X 91 (half sized in each dimension) The result along with ksp view in this case is as follows:
>> 
>> Linear solve converged due to CONVERGED_RTOL iterations 2
>> KSP Object: 64 MPI processes
>>  type: gmres
>>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    GMRES: happy breakdown tolerance 1e-30
>>  maximum iterations=10000, initial guess is zero
>>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>  left preconditioning
>>  has attached null space
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 64 MPI processes
>>  type: fieldsplit
>>    FieldSplit with Schur preconditioner, blocksize = 4, factorization FULL
>>    Preconditioner for the Schur complement formed from user provided matrix
>>    Split info:
>>    Split number 0 Fields  0, 1, 2
>>    Split number 1 Fields  3
>>    KSP solver for A00 block 
>>      KSP Object:      (fieldsplit_0_)       64 MPI processes
>>        type: gmres
>>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>          GMRES: happy breakdown tolerance 1e-30
>>        maximum iterations=10000, initial guess is zero
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>        left preconditioning
>>        using PRECONDITIONED norm type for convergence test
>>      PC Object:      (fieldsplit_0_)       64 MPI processes
>>        type: hypre
>>          HYPRE BoomerAMG preconditioning
>>          HYPRE BoomerAMG: Cycle type V
>>          HYPRE BoomerAMG: Maximum number of levels 25
>>          HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>>          HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>>          HYPRE BoomerAMG: Threshold for strong coupling 0.25
>>          HYPRE BoomerAMG: Interpolation truncation factor 0
>>          HYPRE BoomerAMG: Interpolation: max elements per row 0
>>          HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>>          HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>>          HYPRE BoomerAMG: Maximum row sums 0.9
>>          HYPRE BoomerAMG: Sweeps down         1
>>          HYPRE BoomerAMG: Sweeps up           1
>>          HYPRE BoomerAMG: Sweeps on coarse    1
>>          HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>>          HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>>          HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>>          HYPRE BoomerAMG: Relax weight  (all)      1
>>          HYPRE BoomerAMG: Outer relax weight (all) 1
>>          HYPRE BoomerAMG: Using CF-relaxation
>>          HYPRE BoomerAMG: Measure type        local
>>          HYPRE BoomerAMG: Coarsen type        Falgout
>>          HYPRE BoomerAMG: Interpolation type  classical
>>        linear system matrix = precond matrix:
>>        Matrix Object:         64 MPI processes
>>          type: mpiaij
>>          rows=2793120, cols=2793120
>>          total: nonzeros=221624352, allocated nonzeros=221624352
>>          total number of mallocs used during MatSetValues calls =0
>>            using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>>    KSP solver for S = A11 - A10 inv(A00) A01 
>>      KSP Object:      (fieldsplit_1_)       64 MPI processes
>>        type: gmres
>>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>          GMRES: happy breakdown tolerance 1e-30
>>        maximum iterations=10000, initial guess is zero
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>        left preconditioning
>>        has attached null space
>>        using PRECONDITIONED norm type for convergence test
>>      PC Object:      (fieldsplit_1_)       64 MPI processes
>>        type: bjacobi
>>          block Jacobi: number of blocks = 64
>>          Local solve is same for all blocks, in the following KSP and PC objects:
>>        KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
>>          type: preonly
>>          maximum iterations=10000, initial guess is zero
>>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>          left preconditioning
>>          using NONE norm type for convergence test
>>        PC Object:        (fieldsplit_1_sub_)         1 MPI processes
>>          type: ilu
>>            ILU: out-of-place factorization
>>            0 levels of fill
>>            tolerance for zero pivot 2.22045e-14
>>            using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>            matrix ordering: natural
>>            factor fill ratio given 1, needed 1
>>              Factored matrix follows:
>>                Matrix Object:                 1 MPI processes
>>                  type: seqaij
>>                  rows=14812, cols=14812
>>                  package used to perform factorization: petsc
>>                  total: nonzeros=368098, allocated nonzeros=368098
>>                  total number of mallocs used during MatSetValues calls =0
>>                    not using I-node routines
>>          linear system matrix = precond matrix:
>>          Matrix Object:           1 MPI processes
>>            type: seqaij
>>            rows=14812, cols=14812
>>            total: nonzeros=368098, allocated nonzeros=368098
>>            total number of mallocs used during MatSetValues calls =0
>>              not using I-node routines
>> 
>>        linear system matrix followed by preconditioner matrix:
>>        Matrix Object:         64 MPI processes
>>          type: schurcomplement
>>          rows=931040, cols=931040
>>            Schur complement A11 - A10 inv(A00) A01
>>            A11
>>              Matrix Object:               64 MPI processes
>>                type: mpiaij
>>                rows=931040, cols=931040
>>                total: nonzeros=24624928, allocated nonzeros=24624928
>>                total number of mallocs used during MatSetValues calls =0
>>                  not using I-node (on process 0) routines
>>            A10
>>              Matrix Object:               64 MPI processes
>>                type: mpiaij
>>                rows=931040, cols=2793120
>>                total: nonzeros=73874784, allocated nonzeros=73874784
>>                total number of mallocs used during MatSetValues calls =0
>>                  not using I-node (on process 0) routines
>>            KSP of A00
>>              KSP Object:              (fieldsplit_0_)               64 MPI processes
>>                type: gmres
>>                  GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>                  GMRES: happy breakdown tolerance 1e-30
>>                maximum iterations=10000, initial guess is zero
>>                tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>                left preconditioning
>>                using PRECONDITIONED norm type for convergence test
>>              PC Object:              (fieldsplit_0_)               64 MPI processes
>>                type: hypre
>>                  HYPRE BoomerAMG preconditioning
>>                  HYPRE BoomerAMG: Cycle type V
>>                  HYPRE BoomerAMG: Maximum number of levels 25
>>                  HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>>                  HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>>                  HYPRE BoomerAMG: Threshold for strong coupling 0.25
>>                  HYPRE BoomerAMG: Interpolation truncation factor 0
>>                  HYPRE BoomerAMG: Interpolation: max elements per row 0
>>                  HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>>                  HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>>                  HYPRE BoomerAMG: Maximum row sums 0.9
>>                  HYPRE BoomerAMG: Sweeps down         1
>>                  HYPRE BoomerAMG: Sweeps up           1
>>                  HYPRE BoomerAMG: Sweeps on coarse    1
>>                  HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>>                  HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>>                  HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>>                  HYPRE BoomerAMG: Relax weight  (all)      1
>>                  HYPRE BoomerAMG: Outer relax weight (all) 1
>>                  HYPRE BoomerAMG: Using CF-relaxation
>>                  HYPRE BoomerAMG: Measure type        local
>>                  HYPRE BoomerAMG: Coarsen type        Falgout
>>                  HYPRE BoomerAMG: Interpolation type  classical
>>                linear system matrix = precond matrix:
>>                Matrix Object:                 64 MPI processes
>>                  type: mpiaij
>>                  rows=2793120, cols=2793120
>>                  total: nonzeros=221624352, allocated nonzeros=221624352
>>                  total number of mallocs used during MatSetValues calls =0
>>                    using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>>            A01
>>              Matrix Object:               64 MPI processes
>>                type: mpiaij
>>                rows=2793120, cols=931040
>>                total: nonzeros=73874784, allocated nonzeros=73874784
>>                total number of mallocs used during MatSetValues calls =0
>>                  using I-node (on process 0) routines: found 14812 nodes, limit used is 5
>>        Matrix Object:         64 MPI processes
>>          type: mpiaij
>>          rows=931040, cols=931040
>>          total: nonzeros=24624928, allocated nonzeros=24624928
>>          total number of mallocs used during MatSetValues calls =0
>>            not using I-node (on process 0) routines
>>  linear system matrix = precond matrix:
>>  Matrix Object:   64 MPI processes
>>    type: mpiaij
>>    rows=3724160, cols=3724160, bs=4
>>    total: nonzeros=393998848, allocated nonzeros=393998848
>>    total number of mallocs used during MatSetValues calls =0
>> 
>> ******************************************************************************************************
>> What could be going wrong here ? Is it something related to null-space setting ? But I do not know why it does not arise for smaller domain sizes!
>> 
> 


From bisheshkh at gmail.com  Wed Oct 16 12:55:29 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Wed, 16 Oct 2013 19:55:29 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
Message-ID: <CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>

On Wed, Oct 16, 2013 at 5:50 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

> You might also test with Jacobi as a sanity check.
>
> On Oct 16, 2013, at 11:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> >
> > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> >
> > especially the GAMG version which you also try in the debugger with the
> argument -start_in_debugger
> >
> >
> >
> > On Oct 16, 2013, at 9:32 AM, Dave May <dave.mayhem23 at gmail.com> wrote:
> >
> >> Sounds like a memory error.
> >> I'd run your code through valgrind to double check. The error could be
> completely unconnected to the nullspaces.
>

Thanks, I tried them, but has not worked yet. Here are couple of things I
tried, running valgrind with one and multiple processors for a smaller
sized domains where there is no runtime error thrown. The results are shown
below:
(I've run valgrind in the cluster for the bigger sized domain where it
would throw run-time error, but it's still running, valgrind slowed down
the execution I guess).

1******* ********** For smaller domain sizes ((i.e. the sizes for which the
program runs and gives results) ******************
With one processor, valgrind does NOT give any errors.
With multiple processes it reports something but I'm not sure if they are
errors related to my code. One example with two processes:
petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==31715== Memcheck, a memory error detector
==31716== Memcheck, a memory error detector
==31716== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==31716== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==31716== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==31716==
==31715== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==31715== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==31715== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==31715==
==31716== Conditional jump or move depends on uninitialised value(s)
==31716==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
==31716==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
==31716==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
==31716==    by 0xE: ???
==31716==    by 0x7FF0003EE: ???
==31716==    by 0x7FF0003FC: ???
==31716==    by 0x7FF000405: ???
==31716==    by 0x7FF000410: ???
==31716==    by 0x7FF000424: ???
==31716==
==31716== Conditional jump or move depends on uninitialised value(s)
==31716==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==31716==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
==31716==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
==31716==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
==31716==    by 0xE: ???
==31716==    by 0x7FF0003EE: ???
==31716==    by 0x7FF0003FC: ???
==31716==    by 0x7FF000405: ???
==31716==    by 0x7FF000410: ???
==31716==    by 0x7FF000424: ???
==31716==
dmda of size: (8,8,8)

 using schur complement

 using user defined split
  Linear solve converged due to CONVERGED_ATOL iterations 0
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
Linear solve converged due to CONVERGED_RTOL iterations 1
==31716==
==31716== HEAP SUMMARY:
==31716==     in use at exit: 212,357 bytes in 1,870 blocks
==31716==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
bytes allocated
==31716==
==31715==
==31715== HEAP SUMMARY:
==31715==     in use at exit: 187,709 bytes in 1,864 blocks
==31715==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
bytes allocated
==31715==
==31716== LEAK SUMMARY:
==31716==    definitely lost: 0 bytes in 0 blocks
==31716==    indirectly lost: 0 bytes in 0 blocks
==31716==      possibly lost: 0 bytes in 0 blocks
==31716==    still reachable: 212,357 bytes in 1,870 blocks
==31716==         suppressed: 0 bytes in 0 blocks
==31716== Rerun with --leak-check=full to see details of leaked memory
==31716==
==31716== For counts of detected and suppressed errors, rerun with: -v
==31716== Use --track-origins=yes to see where uninitialised values come
from
==31716== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
==31715== LEAK SUMMARY:
==31715==    definitely lost: 0 bytes in 0 blocks
==31715==    indirectly lost: 0 bytes in 0 blocks
==31715==      possibly lost: 0 bytes in 0 blocks
==31715==    still reachable: 187,709 bytes in 1,864 blocks
==31715==         suppressed: 0 bytes in 0 blocks
==31715== Rerun with --leak-check=full to see details of leaked memory
==31715==
==31715== For counts of detected and suppressed errors, rerun with: -v
==31715== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)



2*************************** For the small size of 18X18X18 grid (the
solver converges, -fieldsplit_0 pc jacobi used) *********************
The results when I run valgrind for 2 processes, some errors:
.....

==18003== Syscall param writev(vector[...]) points to uninitialised byte(s)
==18003==    at 0x962E047: writev (in /lib64/libc-2.14.90.so)
==18003==    by 0xBB34E22: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249)
==18003==    by 0xBB35D52: mca_oob_tcp_peer_send (oob_tcp_peer.c:204)
==18003==    by 0xBB39A36: mca_oob_tcp_send_nb (oob_tcp_send.c:167)
==18003==    by 0xB92AB10: orte_rml_oob_send (rml_oob_send.c:136)
==18003==    by 0xB92B0BF: orte_rml_oob_send_buffer (rml_oob_send.c:270)
==18003==    by 0xBF44147: modex (grpcomm_bad_module.c:573)
==18003==    by 0x81162B1: ompi_mpi_init (ompi_mpi_init.c:541)
==18003==    by 0x812EC31: PMPI_Init_thread (pinit_thread.c:84)
==18003==    by 0x4F903D9: PetscInitialize (pinit.c:675)
==18003==    by 0x505088: main (PetscAdLemMain.cxx:25)
==18003==  Address 0xfdf9d45 is 197 bytes inside a block of size 512 alloc'd
==18003==    at 0x4C2A5B2: realloc (vg_replace_malloc.c:525)
==18003==    by 0x81A4286: opal_dss_buffer_extend
(dss_internal_functions.c:63)
==18003==    by 0x81A4685: opal_dss_copy_payload (dss_load_unload.c:164)
==18003==    by 0x817C07E: orte_grpcomm_base_pack_modex_entries
(grpcomm_base_modex.c:861)
==18003==    by 0xBF44042: modex (grpcomm_bad_module.c:563)
==18003==    by 0x81162B1: ompi_mpi_init (ompi_mpi_init.c:541)
==18003==    by 0x812EC31: PMPI_Init_thread (pinit_thread.c:84)
==18003==    by 0x4F903D9: PetscInitialize (pinit.c:675)
==18003==    by 0x505088: main (PetscAdLemMain.cxx:25)
==18003==

Then the solver converges
and again some errors (I doubt if they are caused from my code at all:)

==18003== Conditional jump or move depends on uninitialised value(s)
==18003==    at 0xDDCC1B2: rdma_destroy_id (in
/usr/lib64/librdmacm.so.1.0.0)
==18003==    by 0xE200A23: id_context_destructor
(btl_openib_connect_rdmacm.c:185)
==18003==    by 0xE1FFED0: rdmacm_component_finalize (opal_object.h:448)
==18003==    by 0xE1FE3AA: ompi_btl_openib_connect_base_finalize
(btl_openib_connect_base.c:496)
==18003==    by 0xE1EA9E6: btl_openib_component_close
(btl_openib_component.c:251)
==18003==    by 0x81BD411: mca_base_components_close
(mca_base_components_close.c:53)
==18003==    by 0x8145E1F: mca_btl_base_close (btl_base_close.c:62)
==18003==    by 0xD5A3DE8: mca_pml_ob1_component_close
(pml_ob1_component.c:156)
==18003==    by 0x81BD411: mca_base_components_close
(mca_base_components_close.c:53)
==18003==    by 0x8154E37: mca_pml_base_close (pml_base_close.c:66)
==18003==    by 0x8117142: ompi_mpi_finalize (ompi_mpi_finalize.c:306)
==18003==    by 0x4F94D6D: PetscFinalize (pinit.c:1276)
==180
..........................

==18003== HEAP SUMMARY:
==18003==     in use at exit: 551,540 bytes in 3,294 blocks
==18003==   total heap usage: 147,859 allocs, 144,565 frees, 84,461,908
bytes allocated
==18003==
==18003== LEAK SUMMARY:
==18003==    definitely lost: 124,956 bytes in 108 blocks
==18003==    indirectly lost: 32,380 bytes in 54 blocks
==18003==      possibly lost: 0 bytes in 0 blocks
==18003==    still reachable: 394,204 bytes in 3,132 blocks
==18003==         suppressed: 0 bytes in 0 blocks
==18003== Rerun with --leak-check=full to see details of leaked memory
==18003==
==18003== For counts of detected and suppressed errors, rerun with: -v
==18003== Use --track-origins=yes to see where uninitialised values come
from
==18003== ERROR SUMMARY: 142 errors from 32 contexts (suppressed: 2 from 2)



> >>
> >> Cheers,
> >>  Dave
> >>
> >>
> >> On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:
> >> Dear all,
> >> I'm trying to solve a stokes flow with constant viscosity but with
> non-zero divergence prescribed in the rhs.
> >>
> >> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p
> respectively.
> >> I have another DMDA (mDaP) of same size but of 1 dof corresponding to
> only p.
> >> I have assigned the null space for constant pressure inside the code. I
> have assigned two nullspace basis:  One corresponding to vector created
> from mDa that is assigned to outer ksp. Second corresponding to vector
> created from mDaP that is assigned to a ksp obtained from the fieldsplit
> corresponding to the schur complement.
> >>
> >> Now when running the code, the solver converges for up to certain size,
> e.g. 92X110 X 92 (the results for this convegent case with -ksp_view is
> given at the end of the emal.
> >> But when I double the size of the grid in each dimension, it gives me a
> run-time error.
> >>
> >> The options  I've used are of the kind:
> >> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits
> 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
> -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
> >>
> >> Here are:
> >> 1. Error message when using hypre for fieldsplit_0
> >> 2. Error message when using gamg for fieldsplit_0
> >> 3. -ksp_view of the working case using hypre for filedsplit_0
> >>
> >> I get following error when I use hypre :
> >> 1.
> ******************************************************************************************************
> >> [5]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> >> [5]PETSC ERROR: Signal received!
> >> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> >> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
> >> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >> [5]PETSC ERROR: See docs/index.html for manual pages.
> >> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [5]PETSC ERROR:
> /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
> arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
> >> [5]PETSC ERROR: Libraries linked from
> /epi/asclepios2/bkhanal/petscDebug/lib
> >> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
> >> [5]PETSC ERROR: Configure options
> --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries
> --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1
> --download-metis --download-parmetis --download-superlu_dist
> --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
> >> [5]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [5]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> >> [6]PETSC ERROR:
> ------------------------------------------------------------------------
> >> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or
> the batch system) has told this process to end
> >> [6]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> >> [6]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> >> [6]PETSC ERROR: likely location of problem given in stack below
> >> [6]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> >> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> >> [6]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> >> [6]PETSC ERROR:       is given.
> >> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130
> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> >> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94
> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
> >> [6]PETSC ERROR: [6] PCSetUp line 868
> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> >> [6]PETSC ERROR: [6] KSPSetUp line 192
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> >> [6]PETSC ERROR: [6] KSPSolve line 356
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> >> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75
> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> >> [6]PETSC ERROR: [6] MatNullSpaceTest line 408
> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> >> [6]PETSC ERROR: [6] solveModel line 113
> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> >>
> >>
> >> 2.
> ****************************************************************************************************
> >> Using gamg instead has errors like following:
> >>
> >> [5]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> >> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> >> [5]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> >> [5]PETSC ERROR:       is given.
> >> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202
> /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
> >> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124
> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> >> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80
> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
> >> [5]PETSC ERROR: [5] MatPtAP line 8223
> /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> >> [5]PETSC ERROR: [5] createLevel line 144
> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> >> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545
> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
> >> [5]PETSC ERROR: [5] PCSetUp line 868
> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
> >> [5]PETSC ERROR: [5] KSPSetUp line 192
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> >> [5]PETSC ERROR: [5] KSPSolve line 356
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> >> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75
> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
> >> [5]PETSC ERROR: [5] MatNullSpaceTest line 408
> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
> >> [5]PETSC ERROR: [5] solveModel line 113
> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> >>
> >>
> >> 3.
> ********************************************************************************************************
> >>
> >> BUT, It does give me results when I use a domain of size: 91X109 X 91
> (half sized in each dimension) The result along with ksp view in this case
> is as follows:
> >>
> >> Linear solve converged due to CONVERGED_RTOL iterations 2
> >> KSP Object: 64 MPI processes
> >>  type: gmres
> >>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >>    GMRES: happy breakdown tolerance 1e-30
> >>  maximum iterations=10000, initial guess is zero
> >>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >>  left preconditioning
> >>  has attached null space
> >>  using PRECONDITIONED norm type for convergence test
> >> PC Object: 64 MPI processes
> >>  type: fieldsplit
> >>    FieldSplit with Schur preconditioner, blocksize = 4, factorization
> FULL
> >>    Preconditioner for the Schur complement formed from user provided
> matrix
> >>    Split info:
> >>    Split number 0 Fields  0, 1, 2
> >>    Split number 1 Fields  3
> >>    KSP solver for A00 block
> >>      KSP Object:      (fieldsplit_0_)       64 MPI processes
> >>        type: gmres
> >>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >>          GMRES: happy breakdown tolerance 1e-30
> >>        maximum iterations=10000, initial guess is zero
> >>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >>        left preconditioning
> >>        using PRECONDITIONED norm type for convergence test
> >>      PC Object:      (fieldsplit_0_)       64 MPI processes
> >>        type: hypre
> >>          HYPRE BoomerAMG preconditioning
> >>          HYPRE BoomerAMG: Cycle type V
> >>          HYPRE BoomerAMG: Maximum number of levels 25
> >>          HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
> >>          HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
> >>          HYPRE BoomerAMG: Threshold for strong coupling 0.25
> >>          HYPRE BoomerAMG: Interpolation truncation factor 0
> >>          HYPRE BoomerAMG: Interpolation: max elements per row 0
> >>          HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
> >>          HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
> >>          HYPRE BoomerAMG: Maximum row sums 0.9
> >>          HYPRE BoomerAMG: Sweeps down         1
> >>          HYPRE BoomerAMG: Sweeps up           1
> >>          HYPRE BoomerAMG: Sweeps on coarse    1
> >>          HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
> >>          HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
> >>          HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
> >>          HYPRE BoomerAMG: Relax weight  (all)      1
> >>          HYPRE BoomerAMG: Outer relax weight (all) 1
> >>          HYPRE BoomerAMG: Using CF-relaxation
> >>          HYPRE BoomerAMG: Measure type        local
> >>          HYPRE BoomerAMG: Coarsen type        Falgout
> >>          HYPRE BoomerAMG: Interpolation type  classical
> >>        linear system matrix = precond matrix:
> >>        Matrix Object:         64 MPI processes
> >>          type: mpiaij
> >>          rows=2793120, cols=2793120
> >>          total: nonzeros=221624352, allocated nonzeros=221624352
> >>          total number of mallocs used during MatSetValues calls =0
> >>            using I-node (on process 0) routines: found 14812 nodes,
> limit used is 5
> >>    KSP solver for S = A11 - A10 inv(A00) A01
> >>      KSP Object:      (fieldsplit_1_)       64 MPI processes
> >>        type: gmres
> >>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >>          GMRES: happy breakdown tolerance 1e-30
> >>        maximum iterations=10000, initial guess is zero
> >>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >>        left preconditioning
> >>        has attached null space
> >>        using PRECONDITIONED norm type for convergence test
> >>      PC Object:      (fieldsplit_1_)       64 MPI processes
> >>        type: bjacobi
> >>          block Jacobi: number of blocks = 64
> >>          Local solve is same for all blocks, in the following KSP and
> PC objects:
> >>        KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
> >>          type: preonly
> >>          maximum iterations=10000, initial guess is zero
> >>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >>          left preconditioning
> >>          using NONE norm type for convergence test
> >>        PC Object:        (fieldsplit_1_sub_)         1 MPI processes
> >>          type: ilu
> >>            ILU: out-of-place factorization
> >>            0 levels of fill
> >>            tolerance for zero pivot 2.22045e-14
> >>            using diagonal shift on blocks to prevent zero pivot
> [INBLOCKS]
> >>            matrix ordering: natural
> >>            factor fill ratio given 1, needed 1
> >>              Factored matrix follows:
> >>                Matrix Object:                 1 MPI processes
> >>                  type: seqaij
> >>                  rows=14812, cols=14812
> >>                  package used to perform factorization: petsc
> >>                  total: nonzeros=368098, allocated nonzeros=368098
> >>                  total number of mallocs used during MatSetValues calls
> =0
> >>                    not using I-node routines
> >>          linear system matrix = precond matrix:
> >>          Matrix Object:           1 MPI processes
> >>            type: seqaij
> >>            rows=14812, cols=14812
> >>            total: nonzeros=368098, allocated nonzeros=368098
> >>            total number of mallocs used during MatSetValues calls =0
> >>              not using I-node routines
> >>
> >>        linear system matrix followed by preconditioner matrix:
> >>        Matrix Object:         64 MPI processes
> >>          type: schurcomplement
> >>          rows=931040, cols=931040
> >>            Schur complement A11 - A10 inv(A00) A01
> >>            A11
> >>              Matrix Object:               64 MPI processes
> >>                type: mpiaij
> >>                rows=931040, cols=931040
> >>                total: nonzeros=24624928, allocated nonzeros=24624928
> >>                total number of mallocs used during MatSetValues calls =0
> >>                  not using I-node (on process 0) routines
> >>            A10
> >>              Matrix Object:               64 MPI processes
> >>                type: mpiaij
> >>                rows=931040, cols=2793120
> >>                total: nonzeros=73874784, allocated nonzeros=73874784
> >>                total number of mallocs used during MatSetValues calls =0
> >>                  not using I-node (on process 0) routines
> >>            KSP of A00
> >>              KSP Object:              (fieldsplit_0_)               64
> MPI processes
> >>                type: gmres
> >>                  GMRES: restart=30, using Classical (unmodified)
> Gram-Schmidt Orthogonalization with no iterative refinement
> >>                  GMRES: happy breakdown tolerance 1e-30
> >>                maximum iterations=10000, initial guess is zero
> >>                tolerances:  relative=1e-05, absolute=1e-50,
> divergence=10000
> >>                left preconditioning
> >>                using PRECONDITIONED norm type for convergence test
> >>              PC Object:              (fieldsplit_0_)               64
> MPI processes
> >>                type: hypre
> >>                  HYPRE BoomerAMG preconditioning
> >>                  HYPRE BoomerAMG: Cycle type V
> >>                  HYPRE BoomerAMG: Maximum number of levels 25
> >>                  HYPRE BoomerAMG: Maximum number of iterations PER
> hypre call 1
> >>                  HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
> >>                  HYPRE BoomerAMG: Threshold for strong coupling 0.25
> >>                  HYPRE BoomerAMG: Interpolation truncation factor 0
> >>                  HYPRE BoomerAMG: Interpolation: max elements per row 0
> >>                  HYPRE BoomerAMG: Number of levels of aggressive
> coarsening 0
> >>                  HYPRE BoomerAMG: Number of paths for aggressive
> coarsening 1
> >>                  HYPRE BoomerAMG: Maximum row sums 0.9
> >>                  HYPRE BoomerAMG: Sweeps down         1
> >>                  HYPRE BoomerAMG: Sweeps up           1
> >>                  HYPRE BoomerAMG: Sweeps on coarse    1
> >>                  HYPRE BoomerAMG: Relax down
>  symmetric-SOR/Jacobi
> >>                  HYPRE BoomerAMG: Relax up
>  symmetric-SOR/Jacobi
> >>                  HYPRE BoomerAMG: Relax on coarse
> Gaussian-elimination
> >>                  HYPRE BoomerAMG: Relax weight  (all)      1
> >>                  HYPRE BoomerAMG: Outer relax weight (all) 1
> >>                  HYPRE BoomerAMG: Using CF-relaxation
> >>                  HYPRE BoomerAMG: Measure type        local
> >>                  HYPRE BoomerAMG: Coarsen type        Falgout
> >>                  HYPRE BoomerAMG: Interpolation type  classical
> >>                linear system matrix = precond matrix:
> >>                Matrix Object:                 64 MPI processes
> >>                  type: mpiaij
> >>                  rows=2793120, cols=2793120
> >>                  total: nonzeros=221624352, allocated nonzeros=221624352
> >>                  total number of mallocs used during MatSetValues calls
> =0
> >>                    using I-node (on process 0) routines: found 14812
> nodes, limit used is 5
> >>            A01
> >>              Matrix Object:               64 MPI processes
> >>                type: mpiaij
> >>                rows=2793120, cols=931040
> >>                total: nonzeros=73874784, allocated nonzeros=73874784
> >>                total number of mallocs used during MatSetValues calls =0
> >>                  using I-node (on process 0) routines: found 14812
> nodes, limit used is 5
> >>        Matrix Object:         64 MPI processes
> >>          type: mpiaij
> >>          rows=931040, cols=931040
> >>          total: nonzeros=24624928, allocated nonzeros=24624928
> >>          total number of mallocs used during MatSetValues calls =0
> >>            not using I-node (on process 0) routines
> >>  linear system matrix = precond matrix:
> >>  Matrix Object:   64 MPI processes
> >>    type: mpiaij
> >>    rows=3724160, cols=3724160, bs=4
> >>    total: nonzeros=393998848, allocated nonzeros=393998848
> >>    total number of mallocs used during MatSetValues calls =0
> >>
> >>
> ******************************************************************************************************
> >> What could be going wrong here ? Is it something related to null-space
> setting ? But I do not know why it does not arise for smaller domain sizes!
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/3b7bed64/attachment-0001.html>

From knepley at gmail.com  Wed Oct 16 13:01:56 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Oct 2013 13:01:56 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
Message-ID: <CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>

On Wed, Oct 16, 2013 at 12:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:

>
>
>
> On Wed, Oct 16, 2013 at 5:50 PM, Mark F. Adams <mfadams at lbl.gov> wrote:
>
>> You might also test with Jacobi as a sanity check.
>>
>> On Oct 16, 2013, at 11:19 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>> >
>> > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> >
>> > especially the GAMG version which you also try in the debugger with the
>> argument -start_in_debugger
>> >
>> >
>> >
>> > On Oct 16, 2013, at 9:32 AM, Dave May <dave.mayhem23 at gmail.com> wrote:
>> >
>> >> Sounds like a memory error.
>> >> I'd run your code through valgrind to double check. The error could be
>> completely unconnected to the nullspaces.
>>
>
> Thanks, I tried them, but has not worked yet. Here are couple of things I
> tried, running valgrind with one and multiple processors for a smaller
> sized domains where there is no runtime error thrown. The results are shown
> below:
> (I've run valgrind in the cluster for the bigger sized domain where it
> would throw run-time error, but it's still running, valgrind slowed down
> the execution I guess).
>

You can also try running under MPICH, which can be valgrind clean.

   Matt


> 1******* ********** For smaller domain sizes ((i.e. the sizes for which
> the program runs and gives results) ******************
> With one processor, valgrind does NOT give any errors.
> With multiple processes it reports something but I'm not sure if they are
> errors related to my code. One example with two processes:
> petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==31715== Memcheck, a memory error detector
> ==31716== Memcheck, a memory error detector
> ==31716== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
> ==31716== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
> ==31716== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==31716==
> ==31715== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
> ==31715== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
> ==31715== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==31715==
> ==31716== Conditional jump or move depends on uninitialised value(s)
> ==31716==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
> ==31716==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
> ==31716==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
> ==31716==    by 0xE: ???
> ==31716==    by 0x7FF0003EE: ???
> ==31716==    by 0x7FF0003FC: ???
> ==31716==    by 0x7FF000405: ???
> ==31716==    by 0x7FF000410: ???
> ==31716==    by 0x7FF000424: ???
> ==31716==
> ==31716== Conditional jump or move depends on uninitialised value(s)
> ==31716==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==31716==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
> ==31716==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
> ==31716==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
> ==31716==    by 0xE: ???
> ==31716==    by 0x7FF0003EE: ???
> ==31716==    by 0x7FF0003FC: ???
> ==31716==    by 0x7FF000405: ???
> ==31716==    by 0x7FF000410: ???
> ==31716==    by 0x7FF000424: ???
> ==31716==
> dmda of size: (8,8,8)
>
>  using schur complement
>
>  using user defined split
>   Linear solve converged due to CONVERGED_ATOL iterations 0
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
> Linear solve converged due to CONVERGED_RTOL iterations 1
> ==31716==
> ==31716== HEAP SUMMARY:
> ==31716==     in use at exit: 212,357 bytes in 1,870 blocks
> ==31716==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
> bytes allocated
> ==31716==
> ==31715==
> ==31715== HEAP SUMMARY:
> ==31715==     in use at exit: 187,709 bytes in 1,864 blocks
> ==31715==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
> bytes allocated
> ==31715==
> ==31716== LEAK SUMMARY:
> ==31716==    definitely lost: 0 bytes in 0 blocks
> ==31716==    indirectly lost: 0 bytes in 0 blocks
> ==31716==      possibly lost: 0 bytes in 0 blocks
> ==31716==    still reachable: 212,357 bytes in 1,870 blocks
> ==31716==         suppressed: 0 bytes in 0 blocks
> ==31716== Rerun with --leak-check=full to see details of leaked memory
> ==31716==
> ==31716== For counts of detected and suppressed errors, rerun with: -v
> ==31716== Use --track-origins=yes to see where uninitialised values come
> from
> ==31716== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
> ==31715== LEAK SUMMARY:
> ==31715==    definitely lost: 0 bytes in 0 blocks
> ==31715==    indirectly lost: 0 bytes in 0 blocks
> ==31715==      possibly lost: 0 bytes in 0 blocks
> ==31715==    still reachable: 187,709 bytes in 1,864 blocks
> ==31715==         suppressed: 0 bytes in 0 blocks
> ==31715== Rerun with --leak-check=full to see details of leaked memory
> ==31715==
> ==31715== For counts of detected and suppressed errors, rerun with: -v
> ==31715== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
>
>
>
> 2*************************** For the small size of 18X18X18 grid (the
> solver converges, -fieldsplit_0 pc jacobi used) *********************
> The results when I run valgrind for 2 processes, some errors:
> .....
>
> ==18003== Syscall param writev(vector[...]) points to uninitialised byte(s)
> ==18003==    at 0x962E047: writev (in /lib64/libc-2.14.90.so)
> ==18003==    by 0xBB34E22: mca_oob_tcp_msg_send_handler (oob_tcp_msg.c:249)
> ==18003==    by 0xBB35D52: mca_oob_tcp_peer_send (oob_tcp_peer.c:204)
> ==18003==    by 0xBB39A36: mca_oob_tcp_send_nb (oob_tcp_send.c:167)
> ==18003==    by 0xB92AB10: orte_rml_oob_send (rml_oob_send.c:136)
> ==18003==    by 0xB92B0BF: orte_rml_oob_send_buffer (rml_oob_send.c:270)
> ==18003==    by 0xBF44147: modex (grpcomm_bad_module.c:573)
> ==18003==    by 0x81162B1: ompi_mpi_init (ompi_mpi_init.c:541)
> ==18003==    by 0x812EC31: PMPI_Init_thread (pinit_thread.c:84)
> ==18003==    by 0x4F903D9: PetscInitialize (pinit.c:675)
> ==18003==    by 0x505088: main (PetscAdLemMain.cxx:25)
> ==18003==  Address 0xfdf9d45 is 197 bytes inside a block of size 512
> alloc'd
> ==18003==    at 0x4C2A5B2: realloc (vg_replace_malloc.c:525)
> ==18003==    by 0x81A4286: opal_dss_buffer_extend
> (dss_internal_functions.c:63)
> ==18003==    by 0x81A4685: opal_dss_copy_payload (dss_load_unload.c:164)
> ==18003==    by 0x817C07E: orte_grpcomm_base_pack_modex_entries
> (grpcomm_base_modex.c:861)
> ==18003==    by 0xBF44042: modex (grpcomm_bad_module.c:563)
> ==18003==    by 0x81162B1: ompi_mpi_init (ompi_mpi_init.c:541)
> ==18003==    by 0x812EC31: PMPI_Init_thread (pinit_thread.c:84)
> ==18003==    by 0x4F903D9: PetscInitialize (pinit.c:675)
> ==18003==    by 0x505088: main (PetscAdLemMain.cxx:25)
> ==18003==
>
> Then the solver converges
> and again some errors (I doubt if they are caused from my code at all:)
>
> ==18003== Conditional jump or move depends on uninitialised value(s)
> ==18003==    at 0xDDCC1B2: rdma_destroy_id (in
> /usr/lib64/librdmacm.so.1.0.0)
> ==18003==    by 0xE200A23: id_context_destructor
> (btl_openib_connect_rdmacm.c:185)
> ==18003==    by 0xE1FFED0: rdmacm_component_finalize (opal_object.h:448)
> ==18003==    by 0xE1FE3AA: ompi_btl_openib_connect_base_finalize
> (btl_openib_connect_base.c:496)
> ==18003==    by 0xE1EA9E6: btl_openib_component_close
> (btl_openib_component.c:251)
> ==18003==    by 0x81BD411: mca_base_components_close
> (mca_base_components_close.c:53)
> ==18003==    by 0x8145E1F: mca_btl_base_close (btl_base_close.c:62)
> ==18003==    by 0xD5A3DE8: mca_pml_ob1_component_close
> (pml_ob1_component.c:156)
> ==18003==    by 0x81BD411: mca_base_components_close
> (mca_base_components_close.c:53)
> ==18003==    by 0x8154E37: mca_pml_base_close (pml_base_close.c:66)
> ==18003==    by 0x8117142: ompi_mpi_finalize (ompi_mpi_finalize.c:306)
> ==18003==    by 0x4F94D6D: PetscFinalize (pinit.c:1276)
> ==180
> ..........................
>
> ==18003== HEAP SUMMARY:
> ==18003==     in use at exit: 551,540 bytes in 3,294 blocks
> ==18003==   total heap usage: 147,859 allocs, 144,565 frees, 84,461,908
> bytes allocated
> ==18003==
> ==18003== LEAK SUMMARY:
> ==18003==    definitely lost: 124,956 bytes in 108 blocks
> ==18003==    indirectly lost: 32,380 bytes in 54 blocks
> ==18003==      possibly lost: 0 bytes in 0 blocks
> ==18003==    still reachable: 394,204 bytes in 3,132 blocks
> ==18003==         suppressed: 0 bytes in 0 blocks
> ==18003== Rerun with --leak-check=full to see details of leaked memory
> ==18003==
> ==18003== For counts of detected and suppressed errors, rerun with: -v
> ==18003== Use --track-origins=yes to see where uninitialised values come
> from
> ==18003== ERROR SUMMARY: 142 errors from 32 contexts (suppressed: 2 from 2)
>
>
>
>> >>
>> >> Cheers,
>> >>  Dave
>> >>
>> >>
>> >> On 16 October 2013 16:18, Bishesh Khanal <bisheshkh at gmail.com> wrote:
>> >> Dear all,
>> >> I'm trying to solve a stokes flow with constant viscosity but with
>> non-zero divergence prescribed in the rhs.
>> >>
>> >> I have a matrix created from DMDA (mDa) of 4 dofs: vx, vy, vz and p
>> respectively.
>> >> I have another DMDA (mDaP) of same size but of 1 dof corresponding to
>> only p.
>> >> I have assigned the null space for constant pressure inside the code.
>> I have assigned two nullspace basis:  One corresponding to vector created
>> from mDa that is assigned to outer ksp. Second corresponding to vector
>> created from mDaP that is assigned to a ksp obtained from the fieldsplit
>> corresponding to the schur complement.
>> >>
>> >> Now when running the code, the solver converges for up to certain
>> size, e.g. 92X110 X 92 (the results for this convegent case with -ksp_view
>> is given at the end of the emal.
>> >> But when I double the size of the grid in each dimension, it gives me
>> a run-time error.
>> >>
>> >> The options  I've used are of the kind:
>> >> -pc_type fieldsplit -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits
>> 0 -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
>> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
>> -fieldsplit_1_ksp_converged_reason -ksp_converged_reason -ksp_view
>> >>
>> >> Here are:
>> >> 1. Error message when using hypre for fieldsplit_0
>> >> 2. Error message when using gamg for fieldsplit_0
>> >> 3. -ksp_view of the working case using hypre for filedsplit_0
>> >>
>> >> I get following error when I use hypre :
>> >> 1.
>> ******************************************************************************************************
>> >> [5]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> >> [5]PETSC ERROR: Signal received!
>> >> [5]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [5]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
>> >> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
>> >> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> >> [5]PETSC ERROR: See docs/index.html for manual pages.
>> >> [5]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [5]PETSC ERROR:
>> /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
>> arch-linux2-cxx-debug named nef001 by bkhanal Wed Oct 16 15:08:42 2013
>> >> [5]PETSC ERROR: Libraries linked from
>> /epi/asclepios2/bkhanal/petscDebug/lib
>> >> [5]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
>> >> [5]PETSC ERROR: Configure options
>> --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries
>> --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
>> >> [5]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [5]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> >> [6]PETSC ERROR:
>> ------------------------------------------------------------------------
>> >> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or
>> the batch system) has told this process to end
>> >> [6]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> >> [6]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> >> [6]PETSC ERROR: likely location of problem given in stack below
>> >> [6]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> >> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> >> [6]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> >> [6]PETSC ERROR:       is given.
>> >> [6]PETSC ERROR: [6] HYPRE_SetupXXX line 130
>> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> >> [6]PETSC ERROR: [6] PCSetUp_HYPRE line 94
>> /tmp/petsc-3.4.3/src/ksp/pc/impls/hypre/hypre.c
>> >> [6]PETSC ERROR: [6] PCSetUp line 868
>> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> >> [6]PETSC ERROR: [6] KSPSetUp line 192
>> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> >> [6]PETSC ERROR: [6] KSPSolve line 356
>> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> >> [6]PETSC ERROR: [6] MatMult_SchurComplement line 75
>> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> >> [6]PETSC ERROR: [6] MatNullSpaceTest line 408
>> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> >> [6]PETSC ERROR: [6] solveModel line 113
>> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>> >>
>> >>
>> >> 2.
>> ****************************************************************************************************
>> >> Using gamg instead has errors like following:
>> >>
>> >> [5]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> >> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> >> [5]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> >> [5]PETSC ERROR:       is given.
>> >> [5]PETSC ERROR: [5] PetscLLCondensedAddSorted line 1202
>> /tmp/petsc-3.4.3/include/petsc-private/matimpl.h
>> >> [5]PETSC ERROR: [5] MatPtAPSymbolic_MPIAIJ_MPIAIJ line 124
>> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> >> [5]PETSC ERROR: [5] MatPtAP_MPIAIJ_MPIAIJ line 80
>> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiptap.c
>> >> [5]PETSC ERROR: [5] MatPtAP line 8223
>> /tmp/petsc-3.4.3/src/mat/interface/matrix.c
>> >> [5]PETSC ERROR: [5] createLevel line 144
>> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> >> [5]PETSC ERROR: [5] PCSetUp_GAMG line 545
>> /tmp/petsc-3.4.3/src/ksp/pc/impls/gamg/gamg.c
>> >> [5]PETSC ERROR: [5] PCSetUp line 868
>> /tmp/petsc-3.4.3/src/ksp/pc/interface/precon.c
>> >> [5]PETSC ERROR: [5] KSPSetUp line 192
>> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> >> [5]PETSC ERROR: [5] KSPSolve line 356
>> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
>> >> [5]PETSC ERROR: [5] MatMult_SchurComplement line 75
>> /tmp/petsc-3.4.3/src/ksp/ksp/utils/schurm.c
>> >> [5]PETSC ERROR: [5] MatNullSpaceTest line 408
>> /tmp/petsc-3.4.3/src/mat/interface/matnull.c
>> >> [5]PETSC ERROR: [5] solveModel line 113
>> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
>> >>
>> >>
>> >> 3.
>> ********************************************************************************************************
>> >>
>> >> BUT, It does give me results when I use a domain of size: 91X109 X 91
>> (half sized in each dimension) The result along with ksp view in this case
>> is as follows:
>> >>
>> >> Linear solve converged due to CONVERGED_RTOL iterations 2
>> >> KSP Object: 64 MPI processes
>> >>  type: gmres
>> >>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >>    GMRES: happy breakdown tolerance 1e-30
>> >>  maximum iterations=10000, initial guess is zero
>> >>  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >>  left preconditioning
>> >>  has attached null space
>> >>  using PRECONDITIONED norm type for convergence test
>> >> PC Object: 64 MPI processes
>> >>  type: fieldsplit
>> >>    FieldSplit with Schur preconditioner, blocksize = 4, factorization
>> FULL
>> >>    Preconditioner for the Schur complement formed from user provided
>> matrix
>> >>    Split info:
>> >>    Split number 0 Fields  0, 1, 2
>> >>    Split number 1 Fields  3
>> >>    KSP solver for A00 block
>> >>      KSP Object:      (fieldsplit_0_)       64 MPI processes
>> >>        type: gmres
>> >>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >>          GMRES: happy breakdown tolerance 1e-30
>> >>        maximum iterations=10000, initial guess is zero
>> >>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >>        left preconditioning
>> >>        using PRECONDITIONED norm type for convergence test
>> >>      PC Object:      (fieldsplit_0_)       64 MPI processes
>> >>        type: hypre
>> >>          HYPRE BoomerAMG preconditioning
>> >>          HYPRE BoomerAMG: Cycle type V
>> >>          HYPRE BoomerAMG: Maximum number of levels 25
>> >>          HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
>> >>          HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
>> >>          HYPRE BoomerAMG: Threshold for strong coupling 0.25
>> >>          HYPRE BoomerAMG: Interpolation truncation factor 0
>> >>          HYPRE BoomerAMG: Interpolation: max elements per row 0
>> >>          HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
>> >>          HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
>> >>          HYPRE BoomerAMG: Maximum row sums 0.9
>> >>          HYPRE BoomerAMG: Sweeps down         1
>> >>          HYPRE BoomerAMG: Sweeps up           1
>> >>          HYPRE BoomerAMG: Sweeps on coarse    1
>> >>          HYPRE BoomerAMG: Relax down          symmetric-SOR/Jacobi
>> >>          HYPRE BoomerAMG: Relax up            symmetric-SOR/Jacobi
>> >>          HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
>> >>          HYPRE BoomerAMG: Relax weight  (all)      1
>> >>          HYPRE BoomerAMG: Outer relax weight (all) 1
>> >>          HYPRE BoomerAMG: Using CF-relaxation
>> >>          HYPRE BoomerAMG: Measure type        local
>> >>          HYPRE BoomerAMG: Coarsen type        Falgout
>> >>          HYPRE BoomerAMG: Interpolation type  classical
>> >>        linear system matrix = precond matrix:
>> >>        Matrix Object:         64 MPI processes
>> >>          type: mpiaij
>> >>          rows=2793120, cols=2793120
>> >>          total: nonzeros=221624352, allocated nonzeros=221624352
>> >>          total number of mallocs used during MatSetValues calls =0
>> >>            using I-node (on process 0) routines: found 14812 nodes,
>> limit used is 5
>> >>    KSP solver for S = A11 - A10 inv(A00) A01
>> >>      KSP Object:      (fieldsplit_1_)       64 MPI processes
>> >>        type: gmres
>> >>          GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> >>          GMRES: happy breakdown tolerance 1e-30
>> >>        maximum iterations=10000, initial guess is zero
>> >>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >>        left preconditioning
>> >>        has attached null space
>> >>        using PRECONDITIONED norm type for convergence test
>> >>      PC Object:      (fieldsplit_1_)       64 MPI processes
>> >>        type: bjacobi
>> >>          block Jacobi: number of blocks = 64
>> >>          Local solve is same for all blocks, in the following KSP and
>> PC objects:
>> >>        KSP Object:        (fieldsplit_1_sub_)         1 MPI processes
>> >>          type: preonly
>> >>          maximum iterations=10000, initial guess is zero
>> >>          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> >>          left preconditioning
>> >>          using NONE norm type for convergence test
>> >>        PC Object:        (fieldsplit_1_sub_)         1 MPI processes
>> >>          type: ilu
>> >>            ILU: out-of-place factorization
>> >>            0 levels of fill
>> >>            tolerance for zero pivot 2.22045e-14
>> >>            using diagonal shift on blocks to prevent zero pivot
>> [INBLOCKS]
>> >>            matrix ordering: natural
>> >>            factor fill ratio given 1, needed 1
>> >>              Factored matrix follows:
>> >>                Matrix Object:                 1 MPI processes
>> >>                  type: seqaij
>> >>                  rows=14812, cols=14812
>> >>                  package used to perform factorization: petsc
>> >>                  total: nonzeros=368098, allocated nonzeros=368098
>> >>                  total number of mallocs used during MatSetValues
>> calls =0
>> >>                    not using I-node routines
>> >>          linear system matrix = precond matrix:
>> >>          Matrix Object:           1 MPI processes
>> >>            type: seqaij
>> >>            rows=14812, cols=14812
>> >>            total: nonzeros=368098, allocated nonzeros=368098
>> >>            total number of mallocs used during MatSetValues calls =0
>> >>              not using I-node routines
>> >>
>> >>        linear system matrix followed by preconditioner matrix:
>> >>        Matrix Object:         64 MPI processes
>> >>          type: schurcomplement
>> >>          rows=931040, cols=931040
>> >>            Schur complement A11 - A10 inv(A00) A01
>> >>            A11
>> >>              Matrix Object:               64 MPI processes
>> >>                type: mpiaij
>> >>                rows=931040, cols=931040
>> >>                total: nonzeros=24624928, allocated nonzeros=24624928
>> >>                total number of mallocs used during MatSetValues calls
>> =0
>> >>                  not using I-node (on process 0) routines
>> >>            A10
>> >>              Matrix Object:               64 MPI processes
>> >>                type: mpiaij
>> >>                rows=931040, cols=2793120
>> >>                total: nonzeros=73874784, allocated nonzeros=73874784
>> >>                total number of mallocs used during MatSetValues calls
>> =0
>> >>                  not using I-node (on process 0) routines
>> >>            KSP of A00
>> >>              KSP Object:              (fieldsplit_0_)               64
>> MPI processes
>> >>                type: gmres
>> >>                  GMRES: restart=30, using Classical (unmodified)
>> Gram-Schmidt Orthogonalization with no iterative refinement
>> >>                  GMRES: happy breakdown tolerance 1e-30
>> >>                maximum iterations=10000, initial guess is zero
>> >>                tolerances:  relative=1e-05, absolute=1e-50,
>> divergence=10000
>> >>                left preconditioning
>> >>                using PRECONDITIONED norm type for convergence test
>> >>              PC Object:              (fieldsplit_0_)               64
>> MPI processes
>> >>                type: hypre
>> >>                  HYPRE BoomerAMG preconditioning
>> >>                  HYPRE BoomerAMG: Cycle type V
>> >>                  HYPRE BoomerAMG: Maximum number of levels 25
>> >>                  HYPRE BoomerAMG: Maximum number of iterations PER
>> hypre call 1
>> >>                  HYPRE BoomerAMG: Convergence tolerance PER hypre call
>> 0
>> >>                  HYPRE BoomerAMG: Threshold for strong coupling 0.25
>> >>                  HYPRE BoomerAMG: Interpolation truncation factor 0
>> >>                  HYPRE BoomerAMG: Interpolation: max elements per row 0
>> >>                  HYPRE BoomerAMG: Number of levels of aggressive
>> coarsening 0
>> >>                  HYPRE BoomerAMG: Number of paths for aggressive
>> coarsening 1
>> >>                  HYPRE BoomerAMG: Maximum row sums 0.9
>> >>                  HYPRE BoomerAMG: Sweeps down         1
>> >>                  HYPRE BoomerAMG: Sweeps up           1
>> >>                  HYPRE BoomerAMG: Sweeps on coarse    1
>> >>                  HYPRE BoomerAMG: Relax down
>>  symmetric-SOR/Jacobi
>> >>                  HYPRE BoomerAMG: Relax up
>>  symmetric-SOR/Jacobi
>> >>                  HYPRE BoomerAMG: Relax on coarse
>> Gaussian-elimination
>> >>                  HYPRE BoomerAMG: Relax weight  (all)      1
>> >>                  HYPRE BoomerAMG: Outer relax weight (all) 1
>> >>                  HYPRE BoomerAMG: Using CF-relaxation
>> >>                  HYPRE BoomerAMG: Measure type        local
>> >>                  HYPRE BoomerAMG: Coarsen type        Falgout
>> >>                  HYPRE BoomerAMG: Interpolation type  classical
>> >>                linear system matrix = precond matrix:
>> >>                Matrix Object:                 64 MPI processes
>> >>                  type: mpiaij
>> >>                  rows=2793120, cols=2793120
>> >>                  total: nonzeros=221624352, allocated
>> nonzeros=221624352
>> >>                  total number of mallocs used during MatSetValues
>> calls =0
>> >>                    using I-node (on process 0) routines: found 14812
>> nodes, limit used is 5
>> >>            A01
>> >>              Matrix Object:               64 MPI processes
>> >>                type: mpiaij
>> >>                rows=2793120, cols=931040
>> >>                total: nonzeros=73874784, allocated nonzeros=73874784
>> >>                total number of mallocs used during MatSetValues calls
>> =0
>> >>                  using I-node (on process 0) routines: found 14812
>> nodes, limit used is 5
>> >>        Matrix Object:         64 MPI processes
>> >>          type: mpiaij
>> >>          rows=931040, cols=931040
>> >>          total: nonzeros=24624928, allocated nonzeros=24624928
>> >>          total number of mallocs used during MatSetValues calls =0
>> >>            not using I-node (on process 0) routines
>> >>  linear system matrix = precond matrix:
>> >>  Matrix Object:   64 MPI processes
>> >>    type: mpiaij
>> >>    rows=3724160, cols=3724160, bs=4
>> >>    total: nonzeros=393998848, allocated nonzeros=393998848
>> >>    total number of mallocs used during MatSetValues calls =0
>> >>
>> >>
>> ******************************************************************************************************
>> >> What could be going wrong here ? Is it something related to null-space
>> setting ? But I do not know why it does not arise for smaller domain sizes!
>> >>
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/08cde466/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed Oct 16 13:02:30 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 13:02:30 -0500
Subject: [petsc-users] extract vector for a 2-D plane from a DM vector
	for a 3-D domain
In-Reply-To: <BAY179-W1598052A1FEADA1CFAD00CB4040@phx.gbl>
References: <BAY179-W88D3794B2D382B8E674711B4040@phx.gbl>
	<87fvs1mgee.fsf@mcs.anl.gov>
	<BAY179-W6247DB7F23AE22470365D6B4040@phx.gbl>
	<871u3lmfc9.fsf@mcs.anl.gov>
	<BAY179-W1598052A1FEADA1CFAD00CB4040@phx.gbl>
Message-ID: <87bo2pksax.fsf@mcs.anl.gov>

Roc Wang <pengxwang at hotmail.com> writes:
> I studied some examples for VecScatter.  To my understand, the pathway
> of extracting the plane is through extracting indexes (IS idx_to) of
> the 2-D plane from the vector of the 3-D domain(IS idx_from). To do
> this, I have to create an array of index for IS idx_to and create the
> corresponding IS by ISCreateGeneral(). Am I right?  Thanks for your
> informative response.

Yes.  You can use DMDAGetAO() for index translation if needed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/a979fd53/attachment.pgp>

From balay at mcs.anl.gov  Wed Oct 16 13:04:15 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 16 Oct 2013 13:04:15 -0500 (CDT)
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>

On Wed, 16 Oct 2013, Matthew Knepley wrote:

> You can also try running under MPICH, which can be valgrind clean.

Actually --download-mpich would configure/install mpich with appropriate flags to be valgrind clean.

Satish

From einar.sorheim at gmail.com  Wed Oct 16 13:22:46 2013
From: einar.sorheim at gmail.com (=?ISO-8859-1?Q?Einar_S=F8rheim?=)
Date: Wed, 16 Oct 2013 20:22:46 +0200
Subject: [petsc-users] Problem with GAMG elasticity case:'createdefaultdata'
 not set(?) need to support NULL data!
Message-ID: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>

Hi
I am testing out the petsc ksp solvers in our own FEM code, in particluar
we are interseted in the GAMG preconditioner.
Applying it to our mechanical problem we get some error messages from
petsc, which I suspect is due to some necessary init calls that are
missing. First the error message:
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data!
[0]PETSC ERROR: 'createdefaultdata' not set(?) need to support NULL data!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-opt named CMP3 by
admeinar Wed Oct 16 18:23:32 2013
[0]PETSC ERROR: Libraries linked from
/cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-opt/lib
[0]PETSC ERROR: Configure run at Wed Oct 16 14:46:01 2013
[0]PETSC ERROR: Configure options --with-debugging=0 --with-openmp=yes
--with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
--with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
(x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
--with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
(x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PCSetUp_GAMG() line 609 in
src/ksp/pc/impls/gamg/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\impls\gamg\gamg.c
[0]PETSC ERROR: PCSetUp() line 890 in
src/ksp/pc/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\INTERF~1\precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in
src/ksp/ksp/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\ksp\INTERF~1\itfunc.c

In short our code looks like the following:


      call KSPGetPC(ksp,pc,ierr)
      tol = 1.e-20
      call
KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
      select case (i_Precond)

      case ( 3 )
         call PCSetType(pc,PCGAMG,ierr)
!         call PCGAMGInitializePackage()
!         call PCCreate_GAMG(pc,ierr)
      case DEFAULT
         call PCSetType(pc,PCJACOBI,ierr)
      end select

!      call PCSetType(pc,PCJACOBI,ierr)
      select case (i_solver)
      case ( 0 )
         call KSPSetType(ksp,KSPCG,ierr)
      case DEFAULT
         call KSPSetType(ksp,KSPCG,ierr)
      end select
      call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
      if (i_nodedof==3) then
         write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
         call
VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
         call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
         call MatSetNearNullSpace(A,Matnull,ierr)
         call MatNullSpaceDestroy(Matnull,ierr)
         call PetscViewerASCIIOpen(PETSC_COMM_WORLD, "Coordvec.m",
viewer,ierr)
         call PetscViewerSetFormat( viewer, PETSC_VIEWER_ASCII_MATLAB,ierr)
         call VecView(vec_coords,viewer,ierr)
         call PetscViewerDestroy( viewer,ierr )
         call VecDestroy(vec_coords,ierr)
         write(*,*) "Finish setting null space ierr =", ierr
!         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
!         CHKERRQ(ierr)
      end if


!      call KSPSetFromOptions(ksp,ierr)
      call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
      call KSPSetUp(ksp,ierr)
      call PetscTime(t0,ierr)
      write(*,*) "Time setup", t0-t1
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      call KSPSolve(ksp,b,x,ierr)
      call PetscTime(t1,ierr)

-- 
Einar S?rheim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/2180d993/attachment.html>

From knepley at gmail.com  Wed Oct 16 13:28:51 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Oct 2013 13:28:51 -0500
Subject: [petsc-users] Problem with GAMG elasticity
 case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
Message-ID: <CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>

On Wed, Oct 16, 2013 at 1:22 PM, Einar S?rheim <einar.sorheim at gmail.com>wrote:

> Hi
> I am testing out the petsc ksp solvers in our own FEM code, in particluar
> we are interseted in the GAMG preconditioner.
> Applying it to our mechanical problem we get some error messages from
> petsc, which I suspect is due to some necessary init calls that are
> missing. First the error message:
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data!
> [0]PETSC ERROR: 'createdefaultdata' not set(?) need to support NULL data!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
> Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-opt named CMP3 by
> admeinar Wed Oct 16 18:23:32 2013
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-opt/lib
> [0]PETSC ERROR: Configure run at Wed Oct 16 14:46:01 2013
> [0]PETSC ERROR: Configure options --with-debugging=0 --with-openmp=yes
> --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
> --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
> (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
> --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
> (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PCSetUp_GAMG() line 609 in
> src/ksp/pc/impls/gamg/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\impls\gamg\gamg.c
> [0]PETSC ERROR: PCSetUp() line 890 in
> src/ksp/pc/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\INTERF~1\precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in
> src/ksp/ksp/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\ksp\INTERF~1\itfunc.c
>
> In short our code looks like the following:
>
>
>       call KSPGetPC(ksp,pc,ierr)
>       tol = 1.e-20
>       call
> KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>       select case (i_Precond)
>
>       case ( 3 )
>          call PCSetType(pc,PCGAMG,ierr)
> !         call PCGAMGInitializePackage()
> !         call PCCreate_GAMG(pc,ierr)
>       case DEFAULT
>          call PCSetType(pc,PCJACOBI,ierr)
>       end select
>
> !      call PCSetType(pc,PCJACOBI,ierr)
>       select case (i_solver)
>       case ( 0 )
>          call KSPSetType(ksp,KSPCG,ierr)
>       case DEFAULT
>          call KSPSetType(ksp,KSPCG,ierr)
>       end select
>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>       if (i_nodedof==3) then
>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>          call
> VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>          call MatSetNearNullSpace(A,Matnull,ierr)
>          call MatNullSpaceDestroy(Matnull,ierr)
>          call PetscViewerASCIIOpen(PETSC_COMM_WORLD, "Coordvec.m",
> viewer,ierr)
>          call PetscViewerSetFormat( viewer, PETSC_VIEWER_ASCII_MATLAB,ierr)
>          call VecView(vec_coords,viewer,ierr)
>          call PetscViewerDestroy( viewer,ierr )
>          call VecDestroy(vec_coords,ierr)
>          write(*,*) "Finish setting null space ierr =", ierr
> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
> !         CHKERRQ(ierr)
>       end if
>
>
> !      call KSPSetFromOptions(ksp,ierr)
>

It looks like you have not set the type of aggregation. If you uncomment
this SetFromOptions()
it should be done automatically.

   Matt


>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>       call KSPSetUp(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "Time setup", t0-t1
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
>       call KSPSolve(ksp,b,x,ierr)
>       call PetscTime(t1,ierr)
>
> --
> Einar S?rheim
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/90653674/attachment.html>

From mfadams at lbl.gov  Wed Oct 16 15:03:02 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Wed, 16 Oct 2013 16:03:02 -0400
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
Message-ID: <8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>

Damn Matt, you know my code better than I do.

Do we require that users call SetFromOptions? Or PCGAMGSetType in this case. This seems a little bad.

Mark

On Oct 16, 2013, at 2:28 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 16, 2013 at 1:22 PM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> Hi
> I am testing out the petsc ksp solvers in our own FEM code, in particluar we are interseted in the GAMG preconditioner.
> Applying it to our mechanical problem we get some error messages from petsc, which I suspect is due to some necessary init calls that are missing. First the error message:
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data!
> [0]PETSC ERROR: 'createdefaultdata' not set(?) need to support NULL data!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-opt named CMP3 by admeinar Wed Oct 16 18:23:32 2013
> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-opt/lib
> [0]PETSC ERROR: Configure run at Wed Oct 16 14:46:01 2013
> [0]PETSC ERROR: Configure options --with-debugging=0 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PCSetUp_GAMG() line 609 in src/ksp/pc/impls/gamg/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\impls\gamg\gamg.c
> [0]PETSC ERROR: PCSetUp() line 890 in src/ksp/pc/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\INTERF~1\precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\ksp\INTERF~1\itfunc.c
> 
> In short our code looks like the following:
> 
> 
>       call KSPGetPC(ksp,pc,ierr)
>       tol = 1.e-20
>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>       select case (i_Precond)
> 
>       case ( 3 )
>          call PCSetType(pc,PCGAMG,ierr)
> !         call PCGAMGInitializePackage()
> !         call PCCreate_GAMG(pc,ierr)
>       case DEFAULT
>          call PCSetType(pc,PCJACOBI,ierr)
>       end select
> 
> !      call PCSetType(pc,PCJACOBI,ierr)
>       select case (i_solver)
>       case ( 0 )
>          call KSPSetType(ksp,KSPCG,ierr)
>       case DEFAULT
>          call KSPSetType(ksp,KSPCG,ierr)
>       end select
>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>       if (i_nodedof==3) then
>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>          call MatSetNearNullSpace(A,Matnull,ierr)
>          call MatNullSpaceDestroy(Matnull,ierr)
>          call PetscViewerASCIIOpen(PETSC_COMM_WORLD, "Coordvec.m", viewer,ierr)
>          call PetscViewerSetFormat( viewer, PETSC_VIEWER_ASCII_MATLAB,ierr)
>          call VecView(vec_coords,viewer,ierr)
>          call PetscViewerDestroy( viewer,ierr )
>          call VecDestroy(vec_coords,ierr)
>          write(*,*) "Finish setting null space ierr =", ierr
> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
> !         CHKERRQ(ierr)
>       end if
> 
> 
> !      call KSPSetFromOptions(ksp,ierr)
> 
> It looks like you have not set the type of aggregation. If you uncomment this SetFromOptions()
> it should be done automatically.
> 
>    Matt
>  
>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>       call KSPSetUp(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "Time setup", t0-t1
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> 
>       call KSPSolve(ksp,b,x,ierr)
>       call PetscTime(t1,ierr) 
> 
> -- 
> Einar S?rheim
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/d00a9f01/attachment-0001.html>

From knepley at gmail.com  Wed Oct 16 15:07:45 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 Oct 2013 15:07:45 -0500
Subject: [petsc-users] Problem with GAMG elasticity
 case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
Message-ID: <CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>

On Wed, Oct 16, 2013 at 3:03 PM, Mark F. Adams <mfadams at lbl.gov> wrote:

> Damn Matt, you know my code better than I do.
>
> Do we require that users call SetFromOptions? Or PCGAMGSetType in this
> case. This seems a little bad.
>

I would normally say that it should have a default GAMGType on creation.

    Matt


> Mark
>
> On Oct 16, 2013, at 2:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Wed, Oct 16, 2013 at 1:22 PM, Einar S?rheim <einar.sorheim at gmail.com>wrote:
>
>> Hi
>> I am testing out the petsc ksp solvers in our own FEM code, in particluar
>> we are interseted in the GAMG preconditioner.
>> Applying it to our mechanical problem we get some error messages from
>> petsc, which I suspect is due to some necessary init calls that are
>> missing. First the error message:
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data!
>> [0]PETSC ERROR: 'createdefaultdata' not set(?) need to support NULL data!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
>> Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-opt named CMP3 by
>> admeinar Wed Oct 16 18:23:32 2013
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-opt/lib
>> [0]PETSC ERROR: Configure run at Wed Oct 16 14:46:01 2013
>> [0]PETSC ERROR: Configure options --with-debugging=0 --with-openmp=yes
>> --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
>> --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
>> (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
>> --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
>> (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: PCSetUp_GAMG() line 609 in
>> src/ksp/pc/impls/gamg/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\impls\gamg\gamg.c
>> [0]PETSC ERROR: PCSetUp() line 890 in
>> src/ksp/pc/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\INTERF~1\precon.c
>> [0]PETSC ERROR: KSPSetUp() line 278 in
>> src/ksp/ksp/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\ksp\INTERF~1\itfunc.c
>>
>> In short our code looks like the following:
>>
>>
>>       call KSPGetPC(ksp,pc,ierr)
>>       tol = 1.e-20
>>       call
>> KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>>       select case (i_Precond)
>>
>>       case ( 3 )
>>          call PCSetType(pc,PCGAMG,ierr)
>> !         call PCGAMGInitializePackage()
>> !         call PCCreate_GAMG(pc,ierr)
>>       case DEFAULT
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       end select
>>
>> !      call PCSetType(pc,PCJACOBI,ierr)
>>       select case (i_solver)
>>       case ( 0 )
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       case DEFAULT
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       end select
>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>>       if (i_nodedof==3) then
>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>>          call
>> VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>>          call MatSetNearNullSpace(A,Matnull,ierr)
>>          call MatNullSpaceDestroy(Matnull,ierr)
>>          call PetscViewerASCIIOpen(PETSC_COMM_WORLD, "Coordvec.m",
>> viewer,ierr)
>>          call PetscViewerSetFormat( viewer,
>> PETSC_VIEWER_ASCII_MATLAB,ierr)
>>          call VecView(vec_coords,viewer,ierr)
>>          call PetscViewerDestroy( viewer,ierr )
>>          call VecDestroy(vec_coords,ierr)
>>          write(*,*) "Finish setting null space ierr =", ierr
>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>> !         CHKERRQ(ierr)
>>       end if
>>
>>
>> !      call KSPSetFromOptions(ksp,ierr)
>>
>
> It looks like you have not set the type of aggregation. If you uncomment
> this SetFromOptions()
> it should be done automatically.
>
>    Matt
>
>
>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>       call KSPSetUp(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Time setup", t0-t1
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> !                      Solve the linear system
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>
>>       call KSPSolve(ksp,b,x,ierr)
>>       call PetscTime(t1,ierr)
>>
>> --
>> Einar S?rheim
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/7452d4ac/attachment.html>

From hsahasra at purdue.edu  Wed Oct 16 19:50:57 2013
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Wed, 16 Oct 2013 20:50:57 -0400 (EDT)
Subject: [petsc-users] MatAij -> MatDense
In-Reply-To: <1760828184.247578.1381970922882.JavaMail.root@mailhub027.itcs.purdue.edu>
Message-ID: <677726772.247586.1381971057219.JavaMail.root@mailhub027.itcs.purdue.edu>

Hi,

Is there any way to convert a sparse matrix to a dense matrix? I have an almost full Aij matrix, which I would like to convert to MatDense and offload it to a GPU.

MatConvert(*mat, MATDENSE, MAT_INITIAL_MATRIX, temp);

mat is of type MATAIJ.

I get the error

No support for this operation for this object type

Thanks!
Harshad


From mfadams at lbl.gov  Wed Oct 16 21:06:53 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Wed, 16 Oct 2013 22:06:53 -0400
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
Message-ID: <1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>


On Oct 16, 2013, at 4:07 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 16, 2013 at 3:03 PM, Mark F. Adams <mfadams at lbl.gov> wrote:
> Damn Matt, you know my code better than I do.
> 
> Do we require that users call SetFromOptions? Or PCGAMGSetType in this case. This seems a little bad.
> 
> I would normally say that it should have a default GAMGType on creation.
> 

I pushed a pull request to fix this.  You should not fail here but you should always use KSPSetFromOptions anyway.

>     Matt
>  
> Mark
> 
> On Oct 16, 2013, at 2:28 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
>> On Wed, Oct 16, 2013 at 1:22 PM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
>> Hi
>> I am testing out the petsc ksp solvers in our own FEM code, in particluar we are interseted in the GAMG preconditioner.
>> Applying it to our mechanical problem we get some error messages from petsc, which I suspect is due to some necessary init calls that are missing. First the error message:
>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data!
>> [0]PETSC ERROR: 'createdefaultdata' not set(?) need to support NULL data!
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-opt named CMP3 by admeinar Wed Oct 16 18:23:32 2013
>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-opt/lib
>> [0]PETSC ERROR: Configure run at Wed Oct 16 14:46:01 2013
>> [0]PETSC ERROR: Configure options --with-debugging=0 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: PCSetUp_GAMG() line 609 in src/ksp/pc/impls/gamg/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\impls\gamg\gamg.c
>> [0]PETSC ERROR: PCSetUp() line 890 in src/ksp/pc/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\pc\INTERF~1\precon.c
>> [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/D:\PROGRA~1\PETSC-~1.3\src\ksp\ksp\INTERF~1\itfunc.c
>> 
>> In short our code looks like the following:
>> 
>> 
>>       call KSPGetPC(ksp,pc,ierr)
>>       tol = 1.e-20
>>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>>       select case (i_Precond)
>> 
>>       case ( 3 )
>>          call PCSetType(pc,PCGAMG,ierr)
>> !         call PCGAMGInitializePackage()
>> !         call PCCreate_GAMG(pc,ierr)
>>       case DEFAULT
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       end select
>> 
>> !      call PCSetType(pc,PCJACOBI,ierr)
>>       select case (i_solver)
>>       case ( 0 )
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       case DEFAULT
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       end select
>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>>       if (i_nodedof==3) then
>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>>          call MatSetNearNullSpace(A,Matnull,ierr)
>>          call MatNullSpaceDestroy(Matnull,ierr)
>>          call PetscViewerASCIIOpen(PETSC_COMM_WORLD, "Coordvec.m", viewer,ierr)
>>          call PetscViewerSetFormat( viewer, PETSC_VIEWER_ASCII_MATLAB,ierr)
>>          call VecView(vec_coords,viewer,ierr)
>>          call PetscViewerDestroy( viewer,ierr )
>>          call VecDestroy(vec_coords,ierr)
>>          write(*,*) "Finish setting null space ierr =", ierr
>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>> !         CHKERRQ(ierr)
>>       end if
>> 
>> 
>> !      call KSPSetFromOptions(ksp,ierr)
>> 
>> It looks like you have not set the type of aggregation. If you uncomment this SetFromOptions()
>> it should be done automatically.
>> 
>>    Matt
>>  
>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>       call KSPSetUp(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Time setup", t0-t1
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>> !                      Solve the linear system
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>> 
>>       call KSPSolve(ksp,b,x,ierr)
>>       call PetscTime(t1,ierr) 
>> 
>> -- 
>> Einar S?rheim
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/0b55ce11/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed Oct 16 21:38:34 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 Oct 2013 21:38:34 -0500
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
Message-ID: <87txggipud.fsf@mcs.anl.gov>

"Mark F. Adams" <mfadams at lbl.gov> writes:
> I pushed a pull request to fix this.  You should not fail here but you
> should always use KSPSetFromOptions anyway.

Amended PR and merged to 'next'.

Mark, relative to your patch, I simplified setting and took out the
now-unnecessary logic in PCSetFromOptions_GAMG.

commit bd94a7aa4cff05a8b49c3b8be9fd5e557f431631
Author: Jed Brown <jedbrown at mcs.anl.gov>
Date:   Wed Oct 16 18:45:43 2013 -0400

    PCGAMG: set default type (AGG); PCSetUp_GAMG crashes if not set
    
    If PCSetFromOptions had not been called, PCSetUp_GAMG would crash due to
    missing subtype.  It is more conventional with top-level objects to use
    a lazy choice of type, but there is no great setup involved, and no
    known order-dependence, so it looks safe to simply set the type to the
    default value (AGG) up-front.
    
    Reported-by: Einar S?rheim <einar.sorheim at gmail.com>
    Based-on-patch-by: Mark Adams <mfadams at lbl.gov>

diff --git a/src/ksp/pc/impls/gamg/gamg.c b/src/ksp/pc/impls/gamg/gamg.c
index 39b3ac4..dadb0e2 100644
--- a/src/ksp/pc/impls/gamg/gamg.c
+++ b/src/ksp/pc/impls/gamg/gamg.c
@@ -1275,12 +1275,11 @@ PetscErrorCode PCSetFromOptions_GAMG(PC pc)
   {
     /* -pc_gamg_type */
     {
-      char tname[256] = PCGAMGAGG;
-      const char *deftype = pc_gamg->gamg_type_name ? pc_gamg->gamg_type_name : tname;
-      ierr = PetscOptionsList("-pc_gamg_type","Type of AMG method","PCGAMGSetType",GAMGList, tname, tname, sizeof(tname), &flag);CHKERRQ(ierr);
+      char tname[256];
+      ierr = PetscOptionsList("-pc_gamg_type","Type of AMG method","PCGAMGSetType",GAMGList, pc_gamg->gamg_type_name, tname, sizeof(tname), &flag);CHKERRQ(ierr);
       /* call PCCreateGAMG_XYZ */
-      if (flag || !pc_gamg->gamg_type_name) {
-        ierr = PCGAMGSetType(pc, flag ? tname : deftype);CHKERRQ(ierr);
+      if (flag) {
+        ierr = PCGAMGSetType(pc,tname);CHKERRQ(ierr);
       }
     }
     /* -pc_gamg_verbose */
@@ -1468,7 +1467,8 @@ PETSC_EXTERN PetscErrorCode PCCreate_GAMG(PC pc)
 #endif
   }
 #endif
-
+  /* PCSetUp_GAMG assumes that the type has been set, so set it to the default now */
+  ierr = PCGAMGSetType(pc,PCGAMGAGG);CHKERRQ(ierr);
   PetscFunctionReturn(0);
 }
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/1600440e/attachment.pgp>

From ckhuangf at gmail.com  Wed Oct 16 23:53:57 2013
From: ckhuangf at gmail.com (Chung-Kan Huang)
Date: Wed, 16 Oct 2013 23:53:57 -0500
Subject: [petsc-users] issues with MatSeqAIJSetPreallocation
Message-ID: <CABRkuZgtTvTVKsj6xiT_djRGyMBJhB3JBfEKbiNgkuaFjYjVug@mail.gmail.com>

Hi,

I am using Petsc to solve a linear system contain variable size of nnz for
each row.
The maximal nnz in some cases could be several hundreds times of minimal
nnz so I was trying to use following,

Mat A;
int N;
PetscInt * d_nnz;
PetscMalloc<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscMalloc.html#PetscMalloc>
((N)*sizeof(PetscInt<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscInt.html#PetscInt>),
&d_nnz);

*** assigning d_nnz ***

MatCreate<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreate.html#MatCreate>
(PETSC_COMM_WORLD<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_COMM_WORLD.html#PETSC_COMM_WORLD>,
&A);
MatSetSizes<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetSizes.html#MatSetSizes>(A,
PETSC_DECIDE<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_DECIDE.html#PETSC_DECIDE>,
PETSC_DECIDE<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PETSC_DECIDE.html#PETSC_DECIDE>,
N, N);
MatSetFromOptions<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetFromOptions.html#MatSetFromOptions>
(A);
MatSeqAIJSetPreallocation<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(A,
0, d_nnz);
PetscFree<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscFree.html#PetscFree>
(d_nnz);

However, by doing this I noticed that it became really slow during
SetValues stage and the reason is that it was allocating memory.


[0]PETSC ERROR: MatSetValues_SeqAIJ() line 353 in
src/mat/impls/aij/seq/aij.c

[0]PETSC ERROR: MatSetValues() line 1106 in src/mat/interface/matrix.c

[0]PETSC ERROR: --------------------- Error Message
------------------------------------

sumintoj: 7980

[0]PETSC ERROR: Argument out of range!

[0]PETSC ERROR: New nonzero at (79800,79800) caused a malloc!


The problem went away after I did
PetscInt d_nz = max(d_nnz);
MatSeqAIJSetPreallocation<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(A,
d_nz,
d_nnz);

But like I mentioned the size of nnz varied and it can be very different
from row to row and it ran out memory pretty quick if the problem is large.

I have checked my d_nnz array and it is correct and  I wonder someone can
point out anything I missed.

Thanks for the help,

Kan

-- 

*Cheers*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131016/40a7d6ad/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 17 00:03:08 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 00:03:08 -0500
Subject: [petsc-users] issues with MatSeqAIJSetPreallocation
In-Reply-To: <CABRkuZgtTvTVKsj6xiT_djRGyMBJhB3JBfEKbiNgkuaFjYjVug@mail.gmail.com>
References: <CABRkuZgtTvTVKsj6xiT_djRGyMBJhB3JBfEKbiNgkuaFjYjVug@mail.gmail.com>
Message-ID: <87hacgij5f.fsf@mcs.anl.gov>

Chung-Kan Huang <ckhuangf at gmail.com> writes:
> [0]PETSC ERROR: Argument out of range!
>
> [0]PETSC ERROR: New nonzero at (79800,79800) caused a malloc!

This means allocation was incorrect.  You should reproduce the problem
with a smaller mesh so that you can debug why it does not work.

> The problem went away after I did
> PetscInt d_nz = max(d_nnz);
> MatSeqAIJSetPreallocation<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(A,
> d_nz,
> d_nnz);

I do not believe this because the first argument (d_nz) is not used when
d_nnz is non-NULL.

> But like I mentioned the size of nnz varied and it can be very different
> from row to row and it ran out memory pretty quick if the problem is large.
>
> I have checked my d_nnz array and it is correct and  I wonder someone can
> point out anything I missed.

Run a smaller problem size in a debugger to find out why you have set
more entries than you expected in that row.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/9d524162/attachment.pgp>

From jedbrown at mcs.anl.gov  Thu Oct 17 00:11:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 00:11:09 -0500
Subject: [petsc-users] MatAij -> MatDense
In-Reply-To: <677726772.247586.1381971057219.JavaMail.root@mailhub027.itcs.purdue.edu>
References: <677726772.247586.1381971057219.JavaMail.root@mailhub027.itcs.purdue.edu>
Message-ID: <878uxsiis2.fsf@mcs.anl.gov>

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:

> Hi,
>
> Is there any way to convert a sparse matrix to a dense matrix? I have an almost full Aij matrix, which I would like to convert to MatDense and offload it to a GPU.
>
> MatConvert(*mat, MATDENSE, MAT_INITIAL_MATRIX, temp);
>
> mat is of type MATAIJ.
>
> I get the error
>
> No support for this operation for this object type

Send the ENTIRE error message.  (ALWAYS send the entire error message.)

There is no special code for the conversion, but MatConvert_Basic should
get called.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/991251b7/attachment.pgp>

From bisheshkh at gmail.com  Thu Oct 17 03:42:33 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 17 Oct 2013 10:42:33 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
Message-ID: <CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>

On Wed, Oct 16, 2013 at 8:04 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Wed, 16 Oct 2013, Matthew Knepley wrote:
>
> > You can also try running under MPICH, which can be valgrind clean.
>
> Actually --download-mpich would configure/install mpich with appropriate
> flags to be valgrind clean.
>

In my laptop (but not in the cluster, please look at the second part of
this reply below for the cluster case) that's how I configured petsc and
ran it under mpich. The following errors (which I do not understand what
they mean) was reported by valgrind when using the mpich of the petsc in my
laptop: Here is the command I used and the error:
(Note: petsc is an alias in my .bashrc: alias
petsc='/home/bkhanal/Documents/softwares/petsc-3.4.3/bin/petscmpiexec'

petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==3106== Memcheck, a memory error detector
==3106== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==3106== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==3107== Memcheck, a memory error detector
==3107== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==3107== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==3107== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==3107==
==3106== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
-pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
-fieldsplit_0_ksp_converged_reason -ksp_converged_reason
==3106==
==3107== Conditional jump or move depends on uninitialised value(s)
==3107==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
==3107==    by 0xE: ???
==3107==    by 0x7FF0003EE: ???
==3107==    by 0x7FF0003FC: ???
==3107==    by 0x7FF000405: ???
==3107==    by 0x7FF000410: ???
==3107==    by 0x7FF000424: ???
==3107==
==3107== Conditional jump or move depends on uninitialised value(s)
==3107==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
==3107==    by 0xE: ???
==3107==    by 0x7FF0003EE: ???
==3107==    by 0x7FF0003FC: ???
==3107==    by 0x7FF000405: ???
==3107==    by 0x7FF000410: ???
==3107==    by 0x7FF000424: ???
==3107==
dmda of size: (8,8,8)

 using schur complement

 using user defined split
  Linear solve converged due to CONVERGED_ATOL iterations 0
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
  Linear solve converged due to CONVERGED_RTOL iterations 3
Linear solve converged due to CONVERGED_RTOL iterations 1
==3106==
==3106== HEAP SUMMARY:
==3106==     in use at exit: 187,709 bytes in 1,864 blocks
==3106==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
bytes allocated
==3106==
==3107==
==3107== HEAP SUMMARY:
==3107==     in use at exit: 212,357 bytes in 1,870 blocks
==3107==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
bytes allocated
==3107==
==3106== LEAK SUMMARY:
==3106==    definitely lost: 0 bytes in 0 blocks
==3106==    indirectly lost: 0 bytes in 0 blocks
==3106==      possibly lost: 0 bytes in 0 blocks
==3106==    still reachable: 187,709 bytes in 1,864 blocks
==3106==         suppressed: 0 bytes in 0 blocks
==3106== Rerun with --leak-check=full to see details of leaked memory
==3106==
==3106== For counts of detected and suppressed errors, rerun with: -v
==3106== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
==3107== LEAK SUMMARY:
==3107==    definitely lost: 0 bytes in 0 blocks
==3107==    indirectly lost: 0 bytes in 0 blocks
==3107==      possibly lost: 0 bytes in 0 blocks
==3107==    still reachable: 212,357 bytes in 1,870 blocks
==3107==         suppressed: 0 bytes in 0 blocks
==3107== Rerun with --leak-check=full to see details of leaked memory
==3107==
==3107== For counts of detected and suppressed errors, rerun with: -v
==3107== Use --track-origins=yes to see where uninitialised values come from
==3107== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)

In the above example, the solver iterates and gives results.

Now the case in cluster: I had to configure petsc with the option:
--with-mpi-dir=/opt/openmpi-gcc/current/  , that's how the cluster
administrators asked me to install to get petsc running in many nodes of
the clusters. I had tried on my own to configure with --download-mpich in
the cluster too, but  could not succeed with some errors. If you really
think the errors could be from this configuration, I would retry to install
with the petscmpich; please let me know.
And the valgrind errors for the case where program terminates without
completing normally (big sized domain), it has following errors just before
abrupt termination:

... lots of other errors and then warnings such as:

==55437== Warning: set address range perms: large range [0xc4369040,
0xd6abb670) (defined)
==55438== Warning: set address range perms: large range [0xc4369040,
0xd6a6cd00) (defined)
==37183== Warning: set address range perms: large range [0xc4369040,
0xd69f57d8) (defined)
==37182== Warning: set address range perms: large range [0xc4369040,
0xd6a474f0) (defined)
mpiexec: killing job...


In between there are several errors such as:
==59334== Use of uninitialised value of size 8
==59334==    at 0xD5B3704: mca_pml_ob1_send_request_put
(pml_ob1_sendreq.c:1217)
==59334==    by 0xE1EF01A: btl_openib_handle_incoming
(btl_openib_component.c:3092)
==59334==    by 0xE1F03E9: btl_openib_component_progress
(btl_openib_component.c:3634)
==59334==    by 0x81CF16A: opal_progress (opal_progress.c:207)
==59334==    by 0x81153AC: ompi_request_default_wait_all (condition.h:92)
==59334==    by 0xF4C25DD: ompi_coll_tuned_sendrecv_actual
(coll_tuned_util.c:54)
==59334==    by 0xF4C91FD:
ompi_coll_tuned_allgatherv_intra_neighborexchange (coll_tuned_util.h:57)
==59334==    by 0x8121783: PMPI_Allgatherv (pallgatherv.c:139)
==59334==    by 0x5156D19: ISAllGather (iscoloring.c:502)
==59334==    by 0x57A6B78: MatGetSubMatrix_MPIAIJ (mpiaij.c:3607)
==59334==    by 0x532DB36: MatGetSubMatrix (matrix.c:7297)
==59334==    by 0x5B97725: PCSetUp_FieldSplit(_p_PC*) (fieldsplit.c:524)






>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/9982695a/attachment-0001.html>

From christophe.ortiz at ciemat.es  Thu Oct 17 05:00:48 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 17 Oct 2013 12:00:48 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
Message-ID: <CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>

On Wed, Oct 16, 2013 at 4:22 PM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
>
> On Wed, Oct 16, 2013 at 4:10 PM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
>
>>
>>
>> On Wed, Oct 16, 2013 at 3:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>>> > I found where is the problem. I had
>>> > set TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE);
>>> > I found it produces wrong solutions, either with ARKIMEX3, 1BEE or A2.
>>> > Solution becomes negative. Without this option, all ARKIMEX types work
>>> very
>>> > well and very fast.
>>>
>>> I'm glad they work as IMEX methods (please check that the solutions are
>>> correct), but I would like to find out why they are not working in fully
>>> implicit mode.  Is your problem similar to ex25?
>>
>>
>> Yes. The only difference is that I put everything (diffusion and reaction
>> terms) under IFunction and IJacobian.
>>
>> My system is
>>
>> u_t - alpha.u_xx + k.u.v = 0
>> v_t - beta.v_xx + k.u.v = 0
>>
>> with Dirichlet boundary conditions
>> u(x=0,t) = u(x=L,t) = cste
>> v(x=0,t) = v(x=L,t) = cste
>>
>>  (If I remember
>>> correctly, that example uses a DAE formulation for boundary conditions
>>> and thus has trouble with some methods with explicit stages.  There may
>>> be more going on.)
>>>
>>
>> Now you say it, I'm having some troubles. After checking that it worked
>> very well with only diffusion terms with ARKIMEX, I introduced the reaction
>> term. With ARKIMEX 1BEE/A2, it goes very fast up to t=0.27 s, then, steps
>> start being rejected and then it goes very slowly, with small timesteps.
>> See output below.
>> Do you think it is due to the fact that I use IFunction to define
>> boundary conditions, which produces troubles with ARKIMEX methods ?
>> What would be the solution ?
>>
>> Christophe
>>
>>
> I just tried with TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE); It goes much
> better and reaches final time....but solution is clearly not correct. The
> initial gaussian distributions do not change. Like if diffusion was null.
>

Hi,

I've tried something else. Since ARKIMEX is able to converge up to t~0.20s
and gives the correct solution for this time, and then comes into trouble,
I thought that it could come from the SNES.
Then, instead of using SNESNEWTONLS, I tried SNESNEWTONTR. I summarize my
findings:

- With NEWTONTR, ARKIMEX (3, 1BEE or A2) converges.

-With NEWTONTR, ARKIMEX 3 converges much faster than 1BEE or A2 (much less
timesteps).

- No conflicts between NEWTONTR and ARKIMEXFullyImplicit as with NEWTONLS.
Give the same results.


Then I came back to SNESNEWTONLS

- With NEWTONLS, ARKIMEX does not converges.

- When ARKIMEXFullyImplicit is used with NEWTONLS, it converges, but
solution clearly incorrect. It seems it does nothing, initial distributions
do not change.


And I played a bit with the tolerances:

- No influence of rtol (increasing or decreasing it). It stops at same
point.

- Changing stol from 1e-8 to 1e-7 seems to ease as it goes up to longer
time (6.98 s instead of 0.20) but then stops with an error:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 450 is not-a-number or infinite
at end of function: Parameter number 3!
...
...
[0]PETSC ERROR: VecValidValues() line 30 in src/vec/vec/interface/rvector.c
[0]PETSC ERROR: SNESComputeFunction() line 1998 in src/snes/interface/snes.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
[0]PETSC ERROR: main() line 392 in src/ts/examples/tutorials/diffusion.c

Maybe all this will give you a hint of what's going on...
Saludos,
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/40a3f639/attachment.html>

From knepley at gmail.com  Thu Oct 17 06:07:39 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Oct 2013 06:07:39 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
Message-ID: <CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>

On Thu, Oct 17, 2013 at 3:42 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Wed, Oct 16, 2013 at 8:04 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> On Wed, 16 Oct 2013, Matthew Knepley wrote:
>>
>> > You can also try running under MPICH, which can be valgrind clean.
>>
>> Actually --download-mpich would configure/install mpich with appropriate
>> flags to be valgrind clean.
>>
>
> In my laptop (but not in the cluster, please look at the second part of
> this reply below for the cluster case) that's how I configured petsc and
> ran it under mpich. The following errors (which I do not understand what
> they mean) was reported by valgrind when using the mpich of the petsc in my
> laptop: Here is the command I used and the error:
>

This is harmless, and as you can see it comes from gfortran initialization.


> (Note: petsc is an alias in my .bashrc: alias
> petsc='/home/bkhanal/Documents/softwares/petsc-3.4.3/bin/petscmpiexec'
>
> petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==3106== Memcheck, a memory error detector
> ==3106== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
> ==3106== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
> ==3107== Memcheck, a memory error detector
> ==3107== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
> ==3107== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
> ==3107== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==3107==
> ==3106== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
> ==3106==
> ==3107== Conditional jump or move depends on uninitialised value(s)
> ==3107==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
> ==3107==    by 0xE: ???
> ==3107==    by 0x7FF0003EE: ???
> ==3107==    by 0x7FF0003FC: ???
> ==3107==    by 0x7FF000405: ???
> ==3107==    by 0x7FF000410: ???
> ==3107==    by 0x7FF000424: ???
> ==3107==
> ==3107== Conditional jump or move depends on uninitialised value(s)
> ==3107==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
> ==3107==    by 0xE: ???
> ==3107==    by 0x7FF0003EE: ???
> ==3107==    by 0x7FF0003FC: ???
> ==3107==    by 0x7FF000405: ???
> ==3107==    by 0x7FF000410: ???
> ==3107==    by 0x7FF000424: ???
> ==3107==
> dmda of size: (8,8,8)
>
>  using schur complement
>
>  using user defined split
>   Linear solve converged due to CONVERGED_ATOL iterations 0
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
>   Linear solve converged due to CONVERGED_RTOL iterations 3
> Linear solve converged due to CONVERGED_RTOL iterations 1
> ==3106==
> ==3106== HEAP SUMMARY:
> ==3106==     in use at exit: 187,709 bytes in 1,864 blocks
> ==3106==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
> bytes allocated
> ==3106==
> ==3107==
> ==3107== HEAP SUMMARY:
> ==3107==     in use at exit: 212,357 bytes in 1,870 blocks
> ==3107==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
> bytes allocated
> ==3107==
> ==3106== LEAK SUMMARY:
> ==3106==    definitely lost: 0 bytes in 0 blocks
> ==3106==    indirectly lost: 0 bytes in 0 blocks
> ==3106==      possibly lost: 0 bytes in 0 blocks
> ==3106==    still reachable: 187,709 bytes in 1,864 blocks
> ==3106==         suppressed: 0 bytes in 0 blocks
> ==3106== Rerun with --leak-check=full to see details of leaked memory
> ==3106==
> ==3106== For counts of detected and suppressed errors, rerun with: -v
> ==3106== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
> ==3107== LEAK SUMMARY:
> ==3107==    definitely lost: 0 bytes in 0 blocks
> ==3107==    indirectly lost: 0 bytes in 0 blocks
> ==3107==      possibly lost: 0 bytes in 0 blocks
> ==3107==    still reachable: 212,357 bytes in 1,870 blocks
> ==3107==         suppressed: 0 bytes in 0 blocks
> ==3107== Rerun with --leak-check=full to see details of leaked memory
> ==3107==
> ==3107== For counts of detected and suppressed errors, rerun with: -v
> ==3107== Use --track-origins=yes to see where uninitialised values come
> from
> ==3107== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
>
> In the above example, the solver iterates and gives results.
>
> Now the case in cluster: I had to configure petsc with the option:
> --with-mpi-dir=/opt/openmpi-gcc/current/  , that's how the cluster
> administrators asked me to install to get petsc running in many nodes of
> the clusters. I had tried on my own to configure with --download-mpich in
> the cluster too, but  could not succeed with some errors. If you really
> think the errors could be from this configuration, I would retry to install
> with the petscmpich; please let me know.
> And the valgrind errors for the case where program terminates without
> completing normally (big sized domain), it has following errors just before
> abrupt termination:
>
> ... lots of other errors and then warnings such as:
>

This appears to be a bug in OpenMPI, which would not be all that
surprising. First, you can try running
in the debugger and extracting a stack trace from the SEGV. Then you could

  1) Get the admin to install MPICH

  2) Try running a PETSc example on the cluster

  3) Try running on another machine

    Matt


> ==55437== Warning: set address range perms: large range [0xc4369040,
> 0xd6abb670) (defined)
> ==55438== Warning: set address range perms: large range [0xc4369040,
> 0xd6a6cd00) (defined)
> ==37183== Warning: set address range perms: large range [0xc4369040,
> 0xd69f57d8) (defined)
> ==37182== Warning: set address range perms: large range [0xc4369040,
> 0xd6a474f0) (defined)
> mpiexec: killing job...
>
>
> In between there are several errors such as:
> ==59334== Use of uninitialised value of size 8
> ==59334==    at 0xD5B3704: mca_pml_ob1_send_request_put
> (pml_ob1_sendreq.c:1217)
> ==59334==    by 0xE1EF01A: btl_openib_handle_incoming
> (btl_openib_component.c:3092)
> ==59334==    by 0xE1F03E9: btl_openib_component_progress
> (btl_openib_component.c:3634)
> ==59334==    by 0x81CF16A: opal_progress (opal_progress.c:207)
> ==59334==    by 0x81153AC: ompi_request_default_wait_all (condition.h:92)
> ==59334==    by 0xF4C25DD: ompi_coll_tuned_sendrecv_actual
> (coll_tuned_util.c:54)
> ==59334==    by 0xF4C91FD:
> ompi_coll_tuned_allgatherv_intra_neighborexchange (coll_tuned_util.h:57)
> ==59334==    by 0x8121783: PMPI_Allgatherv (pallgatherv.c:139)
> ==59334==    by 0x5156D19: ISAllGather (iscoloring.c:502)
> ==59334==    by 0x57A6B78: MatGetSubMatrix_MPIAIJ (mpiaij.c:3607)
> ==59334==    by 0x532DB36: MatGetSubMatrix (matrix.c:7297)
> ==59334==    by 0x5B97725: PCSetUp_FieldSplit(_p_PC*) (fieldsplit.c:524)
>
>
>
>
>
>
>>
>> Satish
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/ce15dc7e/attachment-0001.html>

From christophe.ortiz at ciemat.es  Thu Oct 17 06:56:49 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 17 Oct 2013 13:56:49 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
Message-ID: <CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>

On Thu, Oct 17, 2013 at 12:00 PM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
>
> On Wed, Oct 16, 2013 at 4:22 PM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
>
>>
>>
>> On Wed, Oct 16, 2013 at 4:10 PM, Christophe Ortiz <
>> christophe.ortiz at ciemat.es> wrote:
>>
>>>
>>>
>>> On Wed, Oct 16, 2013 at 3:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>
>>>> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>>>> > I found where is the problem. I had
>>>> > set TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE);
>>>> > I found it produces wrong solutions, either with ARKIMEX3, 1BEE or A2.
>>>> > Solution becomes negative. Without this option, all ARKIMEX types
>>>> work very
>>>> > well and very fast.
>>>>
>>>> I'm glad they work as IMEX methods (please check that the solutions are
>>>> correct), but I would like to find out why they are not working in fully
>>>> implicit mode.  Is your problem similar to ex25?
>>>
>>>
>>> Yes. The only difference is that I put everything (diffusion and
>>> reaction terms) under IFunction and IJacobian.
>>>
>>> My system is
>>>
>>> u_t - alpha.u_xx + k.u.v = 0
>>> v_t - beta.v_xx + k.u.v = 0
>>>
>>> with Dirichlet boundary conditions
>>> u(x=0,t) = u(x=L,t) = cste
>>> v(x=0,t) = v(x=L,t) = cste
>>>
>>>  (If I remember
>>>> correctly, that example uses a DAE formulation for boundary conditions
>>>> and thus has trouble with some methods with explicit stages.  There may
>>>> be more going on.)
>>>>
>>>
>>> Now you say it, I'm having some troubles. After checking that it worked
>>> very well with only diffusion terms with ARKIMEX, I introduced the reaction
>>> term. With ARKIMEX 1BEE/A2, it goes very fast up to t=0.27 s, then, steps
>>> start being rejected and then it goes very slowly, with small timesteps.
>>> See output below.
>>> Do you think it is due to the fact that I use IFunction to define
>>> boundary conditions, which produces troubles with ARKIMEX methods ?
>>> What would be the solution ?
>>>
>>> Christophe
>>>
>>>
>> I just tried with TSARKIMEXSetFullyImplicit(ts,PETSC_TRUE); It goes much
>> better and reaches final time....but solution is clearly not correct. The
>> initial gaussian distributions do not change. Like if diffusion was null.
>>
>
> Hi,
>
> I've tried something else. Since ARKIMEX is able to converge up to t~0.20s
> and gives the correct solution for this time, and then comes into trouble,
> I thought that it could come from the SNES.
> Then, instead of using SNESNEWTONLS, I tried SNESNEWTONTR. I summarize my
> findings:
>
> - With NEWTONTR, ARKIMEX (3, 1BEE or A2) converges.
>
> -With NEWTONTR, ARKIMEX 3 converges much faster than 1BEE or A2 (much less
> timesteps).
>
> - No conflicts between NEWTONTR and ARKIMEXFullyImplicit as with NEWTONLS.
> Give the same results.
>
>
> Then I came back to SNESNEWTONLS
>
> - With NEWTONLS, ARKIMEX does not converges.
>
> - When ARKIMEXFullyImplicit is used with NEWTONLS, it converges, but
> solution clearly incorrect. It seems it does nothing, initial distributions
> do not change.
>
>
> And I played a bit with the tolerances:
>
> - No influence of rtol (increasing or decreasing it). It stops at same
> point.
>
> - Changing stol from 1e-8 to 1e-7 seems to ease as it goes up to longer
> time (6.98 s instead of 0.20) but then stops with an error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Vec entry at local location 450 is not-a-number or
> infinite at end of function: Parameter number 3!
> ...
> ...
> [0]PETSC ERROR: VecValidValues() line 30 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: SNESComputeFunction() line 1998 in
> src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
> [0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
> [0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
> [0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
> [0]PETSC ERROR: main() line 392 in src/ts/examples/tutorials/diffusion.c
>
> Maybe all this will give you a hint of what's going on...
> Saludos,
> Christophe
>
>
>
I've done additional tests:

- NEWTONLS does not work with the default LineSearch, ie bt.

- NEWTONLS works very well with LineSearch basic, l2 and cp. Converges fast
and results are correct.

Seems to be due to the LineSearch bt.

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/989cc847/attachment.html>

From bisheshkh at gmail.com  Thu Oct 17 07:43:17 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 17 Oct 2013 14:43:17 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
Message-ID: <CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>

On Thu, Oct 17, 2013 at 1:07 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 17, 2013 at 3:42 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Wed, Oct 16, 2013 at 8:04 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>
>>> On Wed, 16 Oct 2013, Matthew Knepley wrote:
>>>
>>> > You can also try running under MPICH, which can be valgrind clean.
>>>
>>> Actually --download-mpich would configure/install mpich with appropriate
>>> flags to be valgrind clean.
>>>
>>
>> In my laptop (but not in the cluster, please look at the second part of
>> this reply below for the cluster case) that's how I configured petsc and
>> ran it under mpich. The following errors (which I do not understand what
>> they mean) was reported by valgrind when using the mpich of the petsc in my
>> laptop: Here is the command I used and the error:
>>
>
> This is harmless, and as you can see it comes from gfortran initialization.
>
>
>>  (Note: petsc is an alias in my .bashrc: alias
>> petsc='/home/bkhanal/Documents/softwares/petsc-3.4.3/bin/petscmpiexec'
>>
>> petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>> ==3106== Memcheck, a memory error detector
>> ==3106== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>> ==3106== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>> ==3107== Memcheck, a memory error detector
>> ==3107== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>> ==3107== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
>> ==3107== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>> ==3107==
>> ==3106== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>> ==3106==
>> ==3107== Conditional jump or move depends on uninitialised value(s)
>> ==3107==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0xE: ???
>> ==3107==    by 0x7FF0003EE: ???
>> ==3107==    by 0x7FF0003FC: ???
>> ==3107==    by 0x7FF000405: ???
>> ==3107==    by 0x7FF000410: ???
>> ==3107==    by 0x7FF000424: ???
>> ==3107==
>> ==3107== Conditional jump or move depends on uninitialised value(s)
>> ==3107==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>> ==3107==    by 0xE: ???
>> ==3107==    by 0x7FF0003EE: ???
>> ==3107==    by 0x7FF0003FC: ???
>> ==3107==    by 0x7FF000405: ???
>> ==3107==    by 0x7FF000410: ???
>> ==3107==    by 0x7FF000424: ???
>> ==3107==
>> dmda of size: (8,8,8)
>>
>>  using schur complement
>>
>>  using user defined split
>>   Linear solve converged due to CONVERGED_ATOL iterations 0
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>> Linear solve converged due to CONVERGED_RTOL iterations 1
>> ==3106==
>> ==3106== HEAP SUMMARY:
>> ==3106==     in use at exit: 187,709 bytes in 1,864 blocks
>> ==3106==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
>> bytes allocated
>> ==3106==
>> ==3107==
>> ==3107== HEAP SUMMARY:
>> ==3107==     in use at exit: 212,357 bytes in 1,870 blocks
>> ==3107==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
>> bytes allocated
>> ==3107==
>> ==3106== LEAK SUMMARY:
>> ==3106==    definitely lost: 0 bytes in 0 blocks
>> ==3106==    indirectly lost: 0 bytes in 0 blocks
>> ==3106==      possibly lost: 0 bytes in 0 blocks
>> ==3106==    still reachable: 187,709 bytes in 1,864 blocks
>> ==3106==         suppressed: 0 bytes in 0 blocks
>> ==3106== Rerun with --leak-check=full to see details of leaked memory
>> ==3106==
>> ==3106== For counts of detected and suppressed errors, rerun with: -v
>> ==3106== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
>> ==3107== LEAK SUMMARY:
>> ==3107==    definitely lost: 0 bytes in 0 blocks
>> ==3107==    indirectly lost: 0 bytes in 0 blocks
>> ==3107==      possibly lost: 0 bytes in 0 blocks
>> ==3107==    still reachable: 212,357 bytes in 1,870 blocks
>> ==3107==         suppressed: 0 bytes in 0 blocks
>> ==3107== Rerun with --leak-check=full to see details of leaked memory
>> ==3107==
>> ==3107== For counts of detected and suppressed errors, rerun with: -v
>> ==3107== Use --track-origins=yes to see where uninitialised values come
>> from
>> ==3107== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
>>
>> In the above example, the solver iterates and gives results.
>>
>> Now the case in cluster: I had to configure petsc with the option:
>> --with-mpi-dir=/opt/openmpi-gcc/current/  , that's how the cluster
>> administrators asked me to install to get petsc running in many nodes of
>> the clusters. I had tried on my own to configure with --download-mpich in
>> the cluster too, but  could not succeed with some errors. If you really
>> think the errors could be from this configuration, I would retry to install
>> with the petscmpich; please let me know.
>> And the valgrind errors for the case where program terminates without
>> completing normally (big sized domain), it has following errors just before
>> abrupt termination:
>>
>> ... lots of other errors and then warnings such as:
>>
>
> This appears to be a bug in OpenMPI, which would not be all that
> surprising. First, you can try running
> in the debugger and extracting a stack trace from the SEGV.
>

Just to make sure that I understood what you said before I talk with the
cluster administrators:
The program crashes only for a bigger domain size. Even in the cluster, it
does not crash for the domain size up to a certain size.  So I need to run
in the debugger for the case when it crashes to get the stack trace from
the SEGV, right ? I do not know how to attach a debugger when submitting a
job to the cluster if that is possible at all! Or are you asking me to run
the program in the debugger in my laptop for the biggest size ? (I have not
tried running the code for the biggest size in my laptop fearing it might
take forever)


> Then you could
>
>   1) Get the admin to install MPICH
>
>   2) Try running a PETSc example on the cluster
>

>   3) Try running on another machine
>
>     Matt
>
>
>> ==55437== Warning: set address range perms: large range [0xc4369040,
>> 0xd6abb670) (defined)
>> ==55438== Warning: set address range perms: large range [0xc4369040,
>> 0xd6a6cd00) (defined)
>> ==37183== Warning: set address range perms: large range [0xc4369040,
>> 0xd69f57d8) (defined)
>> ==37182== Warning: set address range perms: large range [0xc4369040,
>> 0xd6a474f0) (defined)
>> mpiexec: killing job...
>>
>>
>> In between there are several errors such as:
>> ==59334== Use of uninitialised value of size 8
>> ==59334==    at 0xD5B3704: mca_pml_ob1_send_request_put
>> (pml_ob1_sendreq.c:1217)
>> ==59334==    by 0xE1EF01A: btl_openib_handle_incoming
>> (btl_openib_component.c:3092)
>> ==59334==    by 0xE1F03E9: btl_openib_component_progress
>> (btl_openib_component.c:3634)
>> ==59334==    by 0x81CF16A: opal_progress (opal_progress.c:207)
>> ==59334==    by 0x81153AC: ompi_request_default_wait_all (condition.h:92)
>> ==59334==    by 0xF4C25DD: ompi_coll_tuned_sendrecv_actual
>> (coll_tuned_util.c:54)
>> ==59334==    by 0xF4C91FD:
>> ompi_coll_tuned_allgatherv_intra_neighborexchange (coll_tuned_util.h:57)
>> ==59334==    by 0x8121783: PMPI_Allgatherv (pallgatherv.c:139)
>> ==59334==    by 0x5156D19: ISAllGather (iscoloring.c:502)
>> ==59334==    by 0x57A6B78: MatGetSubMatrix_MPIAIJ (mpiaij.c:3607)
>> ==59334==    by 0x532DB36: MatGetSubMatrix (matrix.c:7297)
>> ==59334==    by 0x5B97725: PCSetUp_FieldSplit(_p_PC*) (fieldsplit.c:524)
>>
>>
>>
>>
>>
>>
>>>
>>> Satish
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/d2ee8cad/attachment-0001.html>

From knepley at gmail.com  Thu Oct 17 07:46:42 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Oct 2013 07:46:42 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
Message-ID: <CAMYG4G=-Am-O3qDvJ7Yup4P-WGBtWWHA4a3WaU1Hz8veMP=f+A@mail.gmail.com>

On Thu, Oct 17, 2013 at 7:43 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Thu, Oct 17, 2013 at 1:07 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Thu, Oct 17, 2013 at 3:42 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>>
>>>
>>> On Wed, Oct 16, 2013 at 8:04 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>
>>>> On Wed, 16 Oct 2013, Matthew Knepley wrote:
>>>>
>>>> > You can also try running under MPICH, which can be valgrind clean.
>>>>
>>>> Actually --download-mpich would configure/install mpich with
>>>> appropriate flags to be valgrind clean.
>>>>
>>>
>>> In my laptop (but not in the cluster, please look at the second part of
>>> this reply below for the cluster case) that's how I configured petsc and
>>> ran it under mpich. The following errors (which I do not understand what
>>> they mean) was reported by valgrind when using the mpich of the petsc in my
>>> laptop: Here is the command I used and the error:
>>>
>>
>> This is harmless, and as you can see it comes from gfortran
>> initialization.
>>
>>
>>>  (Note: petsc is an alias in my .bashrc: alias
>>> petsc='/home/bkhanal/Documents/softwares/petsc-3.4.3/bin/petscmpiexec'
>>>
>>> petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit
>>> -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0
>>> -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
>>> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
>>> -ksp_converged_reason
>>> ==3106== Memcheck, a memory error detector
>>> ==3106== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>> ==3106== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright
>>> info
>>> ==3107== Memcheck, a memory error detector
>>> ==3107== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>> ==3107== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright
>>> info
>>> ==3107== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>>> ==3107==
>>> ==3106== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>>> ==3106==
>>> ==3107== Conditional jump or move depends on uninitialised value(s)
>>> ==3107==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0xE: ???
>>> ==3107==    by 0x7FF0003EE: ???
>>> ==3107==    by 0x7FF0003FC: ???
>>> ==3107==    by 0x7FF000405: ???
>>> ==3107==    by 0x7FF000410: ???
>>> ==3107==    by 0x7FF000424: ???
>>> ==3107==
>>> ==3107== Conditional jump or move depends on uninitialised value(s)
>>> ==3107==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>>> ==3107==    by 0xE: ???
>>> ==3107==    by 0x7FF0003EE: ???
>>> ==3107==    by 0x7FF0003FC: ???
>>> ==3107==    by 0x7FF000405: ???
>>> ==3107==    by 0x7FF000410: ???
>>> ==3107==    by 0x7FF000424: ???
>>> ==3107==
>>> dmda of size: (8,8,8)
>>>
>>>  using schur complement
>>>
>>>  using user defined split
>>>   Linear solve converged due to CONVERGED_ATOL iterations 0
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>> Linear solve converged due to CONVERGED_RTOL iterations 1
>>> ==3106==
>>> ==3106== HEAP SUMMARY:
>>> ==3106==     in use at exit: 187,709 bytes in 1,864 blocks
>>> ==3106==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
>>> bytes allocated
>>> ==3106==
>>> ==3107==
>>> ==3107== HEAP SUMMARY:
>>> ==3107==     in use at exit: 212,357 bytes in 1,870 blocks
>>> ==3107==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
>>> bytes allocated
>>> ==3107==
>>> ==3106== LEAK SUMMARY:
>>> ==3106==    definitely lost: 0 bytes in 0 blocks
>>> ==3106==    indirectly lost: 0 bytes in 0 blocks
>>> ==3106==      possibly lost: 0 bytes in 0 blocks
>>> ==3106==    still reachable: 187,709 bytes in 1,864 blocks
>>> ==3106==         suppressed: 0 bytes in 0 blocks
>>> ==3106== Rerun with --leak-check=full to see details of leaked memory
>>> ==3106==
>>> ==3106== For counts of detected and suppressed errors, rerun with: -v
>>> ==3106== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
>>> ==3107== LEAK SUMMARY:
>>> ==3107==    definitely lost: 0 bytes in 0 blocks
>>> ==3107==    indirectly lost: 0 bytes in 0 blocks
>>> ==3107==      possibly lost: 0 bytes in 0 blocks
>>> ==3107==    still reachable: 212,357 bytes in 1,870 blocks
>>> ==3107==         suppressed: 0 bytes in 0 blocks
>>> ==3107== Rerun with --leak-check=full to see details of leaked memory
>>> ==3107==
>>> ==3107== For counts of detected and suppressed errors, rerun with: -v
>>> ==3107== Use --track-origins=yes to see where uninitialised values come
>>> from
>>> ==3107== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
>>>
>>> In the above example, the solver iterates and gives results.
>>>
>>> Now the case in cluster: I had to configure petsc with the option:
>>> --with-mpi-dir=/opt/openmpi-gcc/current/  , that's how the cluster
>>> administrators asked me to install to get petsc running in many nodes of
>>> the clusters. I had tried on my own to configure with --download-mpich in
>>> the cluster too, but  could not succeed with some errors. If you really
>>> think the errors could be from this configuration, I would retry to install
>>> with the petscmpich; please let me know.
>>> And the valgrind errors for the case where program terminates without
>>> completing normally (big sized domain), it has following errors just before
>>> abrupt termination:
>>>
>>> ... lots of other errors and then warnings such as:
>>>
>>
>> This appears to be a bug in OpenMPI, which would not be all that
>> surprising. First, you can try running
>> in the debugger and extracting a stack trace from the SEGV.
>>
>
> Just to make sure that I understood what you said before I talk with the
> cluster administrators:
> The program crashes only for a bigger domain size. Even in the cluster, it
> does not crash for the domain size up to a certain size.  So I need to run
> in the debugger for the case when it crashes to get the stack trace from
> the SEGV, right ? I do not know how to attach a debugger when submitting a
> job to the cluster if that is possible at all! Or are you asking me to run
> the program in the debugger in my laptop for the biggest size ? (I have not
> tried running the code for the biggest size in my laptop fearing it might
> take forever)
>

There is no use being afraid. Run it both places. Ask the admins how to map
your display for the cluster and use -start_in_debugger.

   Matt


> Then you could
>>
>>   1) Get the admin to install MPICH
>>
>>   2) Try running a PETSc example on the cluster
>>
>
>>   3) Try running on another machine
>>
>>     Matt
>>
>>
>>> ==55437== Warning: set address range perms: large range [0xc4369040,
>>> 0xd6abb670) (defined)
>>> ==55438== Warning: set address range perms: large range [0xc4369040,
>>> 0xd6a6cd00) (defined)
>>> ==37183== Warning: set address range perms: large range [0xc4369040,
>>> 0xd69f57d8) (defined)
>>> ==37182== Warning: set address range perms: large range [0xc4369040,
>>> 0xd6a474f0) (defined)
>>> mpiexec: killing job...
>>>
>>>
>>> In between there are several errors such as:
>>> ==59334== Use of uninitialised value of size 8
>>> ==59334==    at 0xD5B3704: mca_pml_ob1_send_request_put
>>> (pml_ob1_sendreq.c:1217)
>>> ==59334==    by 0xE1EF01A: btl_openib_handle_incoming
>>> (btl_openib_component.c:3092)
>>> ==59334==    by 0xE1F03E9: btl_openib_component_progress
>>> (btl_openib_component.c:3634)
>>> ==59334==    by 0x81CF16A: opal_progress (opal_progress.c:207)
>>> ==59334==    by 0x81153AC: ompi_request_default_wait_all (condition.h:92)
>>> ==59334==    by 0xF4C25DD: ompi_coll_tuned_sendrecv_actual
>>> (coll_tuned_util.c:54)
>>> ==59334==    by 0xF4C91FD:
>>> ompi_coll_tuned_allgatherv_intra_neighborexchange (coll_tuned_util.h:57)
>>> ==59334==    by 0x8121783: PMPI_Allgatherv (pallgatherv.c:139)
>>> ==59334==    by 0x5156D19: ISAllGather (iscoloring.c:502)
>>> ==59334==    by 0x57A6B78: MatGetSubMatrix_MPIAIJ (mpiaij.c:3607)
>>> ==59334==    by 0x532DB36: MatGetSubMatrix (matrix.c:7297)
>>> ==59334==    by 0x5B97725: PCSetUp_FieldSplit(_p_PC*) (fieldsplit.c:524)
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Satish
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/221efd58/attachment.html>

From bisheshkh at gmail.com  Thu Oct 17 07:52:10 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 17 Oct 2013 14:52:10 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAMYG4G=-Am-O3qDvJ7Yup4P-WGBtWWHA4a3WaU1Hz8veMP=f+A@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<CAMYG4G=-Am-O3qDvJ7Yup4P-WGBtWWHA4a3WaU1Hz8veMP=f+A@mail.gmail.com>
Message-ID: <CAEhex8iV-MA8nqAyExu4xBOJjGiCqAgQRmCWbBmnqGT-zZs=7Q@mail.gmail.com>

On Thu, Oct 17, 2013 at 2:46 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 17, 2013 at 7:43 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Thu, Oct 17, 2013 at 1:07 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Thu, Oct 17, 2013 at 3:42 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Oct 16, 2013 at 8:04 PM, Satish Balay <balay at mcs.anl.gov>wrote:
>>>>
>>>>> On Wed, 16 Oct 2013, Matthew Knepley wrote:
>>>>>
>>>>> > You can also try running under MPICH, which can be valgrind clean.
>>>>>
>>>>> Actually --download-mpich would configure/install mpich with
>>>>> appropriate flags to be valgrind clean.
>>>>>
>>>>
>>>> In my laptop (but not in the cluster, please look at the second part of
>>>> this reply below for the cluster case) that's how I configured petsc and
>>>> ran it under mpich. The following errors (which I do not understand what
>>>> they mean) was reported by valgrind when using the mpich of the petsc in my
>>>> laptop: Here is the command I used and the error:
>>>>
>>>
>>> This is harmless, and as you can see it comes from gfortran
>>> initialization.
>>>
>>>
>>>>  (Note: petsc is an alias in my .bashrc: alias
>>>> petsc='/home/bkhanal/Documents/softwares/petsc-3.4.3/bin/petscmpiexec'
>>>>
>>>> petsc -n 2 valgrind src/AdLemMain -pc_type fieldsplit
>>>> -pc_fieldsplit_type schur -pc_fieldsplit_dm_splits 0
>>>> -pc_fieldsplit_0_fields 0,1,2 -pc_fieldsplit_1_fields 3
>>>> -fieldsplit_0_pc_type hypre -fieldsplit_0_ksp_converged_reason
>>>> -ksp_converged_reason
>>>> ==3106== Memcheck, a memory error detector
>>>> ==3106== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>>> ==3106== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright
>>>> info
>>>> ==3107== Memcheck, a memory error detector
>>>> ==3107== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
>>>> ==3107== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright
>>>> info
>>>> ==3107== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>>>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>>>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>>>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>>>> ==3107==
>>>> ==3106== Command: src/AdLemMain -pc_type fieldsplit -pc_fieldsplit_type
>>>> schur -pc_fieldsplit_dm_splits 0 -pc_fieldsplit_0_fields 0,1,2
>>>> -pc_fieldsplit_1_fields 3 -fieldsplit_0_pc_type hypre
>>>> -fieldsplit_0_ksp_converged_reason -ksp_converged_reason
>>>> ==3106==
>>>> ==3107== Conditional jump or move depends on uninitialised value(s)
>>>> ==3107==    at 0x32EEED9BCE: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0xE: ???
>>>> ==3107==    by 0x7FF0003EE: ???
>>>> ==3107==    by 0x7FF0003FC: ???
>>>> ==3107==    by 0x7FF000405: ???
>>>> ==3107==    by 0x7FF000410: ???
>>>> ==3107==    by 0x7FF000424: ???
>>>> ==3107==
>>>> ==3107== Conditional jump or move depends on uninitialised value(s)
>>>> ==3107==    at 0x32EEED9BD9: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32EEED9155: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32EEE185D7: ??? (in /usr/lib64/libgfortran.so.3.0.0)
>>>> ==3107==    by 0x32ECC0F195: call_init.part.0 (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0x32ECC0F272: _dl_init (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0x32ECC01719: ??? (in /lib64/ld-2.14.90.so)
>>>> ==3107==    by 0xE: ???
>>>> ==3107==    by 0x7FF0003EE: ???
>>>> ==3107==    by 0x7FF0003FC: ???
>>>> ==3107==    by 0x7FF000405: ???
>>>> ==3107==    by 0x7FF000410: ???
>>>> ==3107==    by 0x7FF000424: ???
>>>> ==3107==
>>>> dmda of size: (8,8,8)
>>>>
>>>>  using schur complement
>>>>
>>>>  using user defined split
>>>>   Linear solve converged due to CONVERGED_ATOL iterations 0
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>>   Linear solve converged due to CONVERGED_RTOL iterations 3
>>>> Linear solve converged due to CONVERGED_RTOL iterations 1
>>>> ==3106==
>>>> ==3106== HEAP SUMMARY:
>>>> ==3106==     in use at exit: 187,709 bytes in 1,864 blocks
>>>> ==3106==   total heap usage: 112,891 allocs, 111,027 frees, 19,838,487
>>>> bytes allocated
>>>> ==3106==
>>>> ==3107==
>>>> ==3107== HEAP SUMMARY:
>>>> ==3107==     in use at exit: 212,357 bytes in 1,870 blocks
>>>> ==3107==   total heap usage: 112,701 allocs, 110,831 frees, 19,698,341
>>>> bytes allocated
>>>> ==3107==
>>>> ==3106== LEAK SUMMARY:
>>>> ==3106==    definitely lost: 0 bytes in 0 blocks
>>>> ==3106==    indirectly lost: 0 bytes in 0 blocks
>>>> ==3106==      possibly lost: 0 bytes in 0 blocks
>>>> ==3106==    still reachable: 187,709 bytes in 1,864 blocks
>>>> ==3106==         suppressed: 0 bytes in 0 blocks
>>>> ==3106== Rerun with --leak-check=full to see details of leaked memory
>>>> ==3106==
>>>> ==3106== For counts of detected and suppressed errors, rerun with: -v
>>>> ==3106== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
>>>> ==3107== LEAK SUMMARY:
>>>> ==3107==    definitely lost: 0 bytes in 0 blocks
>>>> ==3107==    indirectly lost: 0 bytes in 0 blocks
>>>> ==3107==      possibly lost: 0 bytes in 0 blocks
>>>> ==3107==    still reachable: 212,357 bytes in 1,870 blocks
>>>> ==3107==         suppressed: 0 bytes in 0 blocks
>>>> ==3107== Rerun with --leak-check=full to see details of leaked memory
>>>> ==3107==
>>>> ==3107== For counts of detected and suppressed errors, rerun with: -v
>>>> ==3107== Use --track-origins=yes to see where uninitialised values come
>>>> from
>>>> ==3107== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from 2)
>>>>
>>>> In the above example, the solver iterates and gives results.
>>>>
>>>> Now the case in cluster: I had to configure petsc with the option:
>>>> --with-mpi-dir=/opt/openmpi-gcc/current/  , that's how the cluster
>>>> administrators asked me to install to get petsc running in many nodes of
>>>> the clusters. I had tried on my own to configure with --download-mpich in
>>>> the cluster too, but  could not succeed with some errors. If you really
>>>> think the errors could be from this configuration, I would retry to install
>>>> with the petscmpich; please let me know.
>>>> And the valgrind errors for the case where program terminates without
>>>> completing normally (big sized domain), it has following errors just before
>>>> abrupt termination:
>>>>
>>>> ... lots of other errors and then warnings such as:
>>>>
>>>
>>> This appears to be a bug in OpenMPI, which would not be all that
>>> surprising. First, you can try running
>>> in the debugger and extracting a stack trace from the SEGV.
>>>
>>
>> Just to make sure that I understood what you said before I talk with the
>> cluster administrators:
>> The program crashes only for a bigger domain size. Even in the cluster,
>> it does not crash for the domain size up to a certain size.  So I need to
>> run in the debugger for the case when it crashes to get the stack trace
>> from the SEGV, right ? I do not know how to attach a debugger when
>> submitting a job to the cluster if that is possible at all! Or are you
>> asking me to run the program in the debugger in my laptop for the biggest
>> size ? (I have not tried running the code for the biggest size in my laptop
>> fearing it might take forever)
>>
>
> There is no use being afraid. Run it both places. Ask the admins how to
> map your display for the cluster and use -start_in_debugger.
>

 Thanks, I'll try them.


>    Matt
>
>
>>  Then you could
>>>
>>>   1) Get the admin to install MPICH
>>>
>>>   2) Try running a PETSc example on the cluster
>>>
>>
>>>   3) Try running on another machine
>>>
>>>     Matt
>>>
>>>
>>>> ==55437== Warning: set address range perms: large range [0xc4369040,
>>>> 0xd6abb670) (defined)
>>>> ==55438== Warning: set address range perms: large range [0xc4369040,
>>>> 0xd6a6cd00) (defined)
>>>> ==37183== Warning: set address range perms: large range [0xc4369040,
>>>> 0xd69f57d8) (defined)
>>>> ==37182== Warning: set address range perms: large range [0xc4369040,
>>>> 0xd6a474f0) (defined)
>>>> mpiexec: killing job...
>>>>
>>>>
>>>> In between there are several errors such as:
>>>> ==59334== Use of uninitialised value of size 8
>>>> ==59334==    at 0xD5B3704: mca_pml_ob1_send_request_put
>>>> (pml_ob1_sendreq.c:1217)
>>>> ==59334==    by 0xE1EF01A: btl_openib_handle_incoming
>>>> (btl_openib_component.c:3092)
>>>> ==59334==    by 0xE1F03E9: btl_openib_component_progress
>>>> (btl_openib_component.c:3634)
>>>> ==59334==    by 0x81CF16A: opal_progress (opal_progress.c:207)
>>>> ==59334==    by 0x81153AC: ompi_request_default_wait_all
>>>> (condition.h:92)
>>>> ==59334==    by 0xF4C25DD: ompi_coll_tuned_sendrecv_actual
>>>> (coll_tuned_util.c:54)
>>>> ==59334==    by 0xF4C91FD:
>>>> ompi_coll_tuned_allgatherv_intra_neighborexchange (coll_tuned_util.h:57)
>>>> ==59334==    by 0x8121783: PMPI_Allgatherv (pallgatherv.c:139)
>>>> ==59334==    by 0x5156D19: ISAllGather (iscoloring.c:502)
>>>> ==59334==    by 0x57A6B78: MatGetSubMatrix_MPIAIJ (mpiaij.c:3607)
>>>> ==59334==    by 0x532DB36: MatGetSubMatrix (matrix.c:7297)
>>>> ==59334==    by 0x5B97725: PCSetUp_FieldSplit(_p_PC*) (fieldsplit.c:524)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Satish
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/fedc8a52/attachment-0001.html>

From jedbrown at mcs.anl.gov  Thu Oct 17 08:00:54 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 08:00:54 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
Message-ID: <87zjq8gigp.fsf@mcs.anl.gov>

Bishesh Khanal <bisheshkh at gmail.com> writes:
> The program crashes only for a bigger domain size. Even in the cluster, it
> does not crash for the domain size up to a certain size.  So I need to run
> in the debugger for the case when it crashes to get the stack trace from
> the SEGV, right ? I do not know how to attach a debugger when submitting a
> job to the cluster if that is possible at all! 

Most machines allow you to get "interactive" sessions.  You can usually
run debuggers within those.  Some facilities also have commercial
debuggers.

> Or are you asking me to run the program in the debugger in my laptop
> for the biggest size ? (I have not tried running the code for the
> biggest size in my laptop fearing it might take forever)

Your laptop probably doesn't have enough memory for that.

Can you try running on the cluster with one MPI rank per node?  We
should rule out simple out-of-memory problems, confirm that the code
executes correctly with MPICH, and finally figure out why it fails with
Open MPI (assuming that the previous hunch was correct).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/63b52905/attachment.pgp>

From bisheshkh at gmail.com  Thu Oct 17 08:47:38 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 17 Oct 2013 15:47:38 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <87zjq8gigp.fsf@mcs.anl.gov>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
Message-ID: <CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>

On Thu, Oct 17, 2013 at 3:00 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Bishesh Khanal <bisheshkh at gmail.com> writes:
> > The program crashes only for a bigger domain size. Even in the cluster,
> it
> > does not crash for the domain size up to a certain size.  So I need to
> run
> > in the debugger for the case when it crashes to get the stack trace from
> > the SEGV, right ? I do not know how to attach a debugger when submitting
> a
> > job to the cluster if that is possible at all!
>
> Most machines allow you to get "interactive" sessions.  You can usually
> run debuggers within those.  Some facilities also have commercial
> debuggers.
>

Thanks, I'll have a look at that.


>
> > Or are you asking me to run the program in the debugger in my laptop
> > for the biggest size ? (I have not tried running the code for the
> > biggest size in my laptop fearing it might take forever)
>
> Your laptop probably doesn't have enough memory for that.
>

Yes, I tried it just a while ago and this is happened I think. (Just to
confirm, I have put the error message for this case at the very end of this
reply.*)


>
> Can you try running on the cluster with one MPI rank per node?  We
> should rule out simple out-of-memory problems, confirm that the code
> executes correctly with MPICH, and finally figure out why it fails with
> Open MPI (assuming that the previous hunch was correct).
>
>
I'm sorry but I'm a complete beginner with MPI and clusters; so what does
one MPI rank per node means and what should I do to do that ? My guess is
that I set one core per node and use multiple nodes in my job script file ?
Or do I need to do something in the petsc code ?

*Here is the error I get when running for the full domain size in my laptop:
[3]PETSC ERROR: --------------------- Error Message
------------------------------------
[3]PETSC ERROR: Out of memory. This could be due to allocating
[3]PETSC ERROR: too large an object or bleeding by not properly
[3]PETSC ERROR: destroying unneeded objects.
[1]PETSC ERROR: Memory allocated 0 Memory used by process 1700159488
[1]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[1]PETSC ERROR: Memory requested 6234924800!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: [2]PETSC ERROR: Memory allocated 0 Memory used by process
1695793152
[2]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[2]PETSC ERROR: Memory requested 6223582208!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
bkhanal Thu Oct 17 15:19:22 2013
[1]PETSC ERROR: Libraries linked from
/home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
[1]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
[1]PETSC ERROR: Configure options --download-mpich
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: PetscMallocAlign() line 46 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
src/AdLemMain on a arch-linux2-cxx-debug named edwards by bkhanal Thu Oct
17 15:19:22 2013
[2]PETSC ERROR: Libraries linked from
/home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
[2]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
[2]PETSC ERROR: Configure options --download-mpich
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: PetscMallocAlign() line 46 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
[1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[1]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[2]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[2]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[1]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[2]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[2]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
[2]PETSC ERROR: [1]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[1]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[2]PETSC ERROR: DMCreateMatrix_DA() line 771 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
DMCreateMatrix_DA() line 771 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[3]PETSC ERROR: Memory allocated 0 Memory used by process 1675407360
[3]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[3]PETSC ERROR: Memory requested 6166659200!
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[3]PETSC ERROR: See docs/changes/index.html for recent updates.
[3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[3]PETSC ERROR: See docs/index.html for manual pages.
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
bkhanal Thu Oct 17 15:19:22 2013
[3]PETSC ERROR: Libraries linked from
/home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
[3]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
[3]PETSC ERROR: Configure options --download-mpich
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[3]PETSC ERROR:
------------------------------------------------------------------------
[3]PETSC ERROR: [1]PETSC ERROR: DMCreateMatrix() line 910 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
[2]PETSC ERROR: DMCreateMatrix() line 910 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
PetscMallocAlign() line 46 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
[3]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[3]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[1]PETSC ERROR: KSPSetUp() line 207 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: KSPSetUp() line 207 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[3]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[3]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[3]PETSC ERROR: DMCreateMatrix_DA() line 771 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[3]PETSC ERROR: DMCreateMatrix() line 910 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
[3]PETSC ERROR: KSPSetUp() line 207 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: solveModel() line 128 in
"unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
[2]PETSC ERROR: solveModel() line 128 in
"unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
[3]PETSC ERROR: solveModel() line 128 in
"unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
[0]PETSC ERROR: Memory allocated 0 Memory used by process 1711476736
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[0]PETSC ERROR: Memory requested 6292477952!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
bkhanal Thu Oct 17 15:19:22 2013
[0]PETSC ERROR: Libraries linked from
/home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
[0]PETSC ERROR: Configure options --download-mpich
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscMallocAlign() line 46 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
[0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[0]PETSC ERROR: DMCreateMatrix_DA() line 771 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
[0]PETSC ERROR: DMCreateMatrix() line 910 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
[0]PETSC ERROR: KSPSetUp() line 207 in
/home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: solveModel() line 128 in
"unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
--9345:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
--9345:0:aspacem    Increase it and rebuild.  Exiting now.
--9344:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
--9344:0:aspacem    Increase it and rebuild.  Exiting now.
--9343:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
--9343:0:aspacem    Increase it and rebuild.  Exiting now.
--9346:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
--9346:0:aspacem    Increase it and rebuild.  Exiting now.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/105dc6ef/attachment.html>

From rudraa at umich.edu  Thu Oct 17 09:15:27 2013
From: rudraa at umich.edu (Shiva Rudraraju)
Date: Thu, 17 Oct 2013 10:15:27 -0400
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
Message-ID: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>

Hi,

I am planning to study the effects of geometric multigrid method both as a
Solver and Preconditioner on a system of transport problems involving
coupled diffusion + finite strain mechanics. Part of that study will be to
evaluate the performance of FD vs higher order FEM (Both Lagrange and
Spectral elements).

So in this context I have a few questions:
1)  To motivate the use of MG schemes in this coupled system of equations,
I plan to initially implement a Red-Black Gauss Siedel solver for FD to
serve as a benchmark. Is there a good example related to the Red-Black
Gauss Siedel implementation?.
.
2) I plan to implement a higher order Lagrange and Spectral element FEM
code. I found one related example in
petsc/src/ksp/ksp/examples/tutorials/ex59.c. Are there any better examples
of general FEM code implementations.

3) MG is often used as a preconditioner, but are there any examples showing
its use as a solver?. I found a few examples here
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/concepts/multigrid.html,
but nothing in the current version of those codes refer to any MG
datastructures like DMMG which were part of the earlier versions.

Thanks,
Shiva Rudraraju
Univ. of Michigan Ann Arbor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/8ef410a9/attachment-0001.html>

From bisheshkh at gmail.com  Thu Oct 17 09:26:19 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 17 Oct 2013 16:26:19 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
	<CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
Message-ID: <CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>

On Thu, Oct 17, 2013 at 3:47 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Thu, Oct 17, 2013 at 3:00 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Bishesh Khanal <bisheshkh at gmail.com> writes:
>> > The program crashes only for a bigger domain size. Even in the cluster,
>> it
>> > does not crash for the domain size up to a certain size.  So I need to
>> run
>> > in the debugger for the case when it crashes to get the stack trace from
>> > the SEGV, right ? I do not know how to attach a debugger when
>> submitting a
>> > job to the cluster if that is possible at all!
>>
>> Most machines allow you to get "interactive" sessions.  You can usually
>> run debuggers within those.  Some facilities also have commercial
>> debuggers.
>>
>
> Thanks, I'll have a look at that.
>
>
>>
>> > Or are you asking me to run the program in the debugger in my laptop
>> > for the biggest size ? (I have not tried running the code for the
>> > biggest size in my laptop fearing it might take forever)
>>
>> Your laptop probably doesn't have enough memory for that.
>>
>
> Yes, I tried it just a while ago and this is happened I think. (Just to
> confirm, I have put the error message for this case at the very end of this
> reply.*)
>
>
>>
>> Can you try running on the cluster with one MPI rank per node?  We
>> should rule out simple out-of-memory problems, confirm that the code
>> executes correctly with MPICH, and finally figure out why it fails with
>> Open MPI (assuming that the previous hunch was correct).
>>
>>
> I tried running on the cluster with one core per node with 4 nodes and I
got the following errors (note: using valgrind, and openmpi of the cluster)
at the very end after the many usual "unconditional jump ... errors"  which
might be interesting

mpiexec: killing job...

mpiexec: abort is already in progress...hit ctrl-c again to forcibly
terminate

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatSetValues_MPIAIJ line 505
/tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: [0] MatSetValues line 1071
/tmp/petsc-3.4.3/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] MatSetValuesLocal line 1935
/tmp/petsc-3.4.3/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] DMCreateMatrix_DA_3d_MPIAIJ line 1051
/tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
[0]PETSC ERROR: [0] DMCreateMatrix_DA line 627
/tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
[0]PETSC ERROR: [0] DMCreateMatrix line 900
/tmp/petsc-3.4.3/src/dm/interface/dm.c
[0]PETSC ERROR: [0] KSPSetUp line 192
/tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] solveModel line 122
"unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR:
/epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
arch-linux2-cxx-debug named nef002 by bkhanal Thu Oct 17 15:55:33 2013
[0]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
[0]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
[0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/
--with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
==47363==
==47363== HEAP SUMMARY:
==47363==     in use at exit: 10,939,838,029 bytes in 8,091 blocks
==47363==   total heap usage: 1,936,963 allocs, 1,928,872 frees,
11,530,164,042 bytes allocated
==47363==

Does it mean it is crashing near MatSetValues_MPIAIJ ?



> I'm sorry but I'm a complete beginner with MPI and clusters; so what does
> one MPI rank per node means and what should I do to do that ? My guess is
> that I set one core per node and use multiple nodes in my job script file ?
> Or do I need to do something in the petsc code ?
>
> *Here is the error I get when running for the full domain size in my
> laptop:
> [3]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [3]PETSC ERROR: Out of memory. This could be due to allocating
> [3]PETSC ERROR: too large an object or bleeding by not properly
> [3]PETSC ERROR: destroying unneeded objects.
> [1]PETSC ERROR: Memory allocated 0 Memory used by process 1700159488
> [1]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [1]PETSC ERROR: Memory requested 6234924800!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: [2]PETSC ERROR: Memory allocated 0 Memory used by process
> 1695793152
> [2]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [2]PETSC ERROR: Memory requested 6223582208!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
> bkhanal Thu Oct 17 15:19:22 2013
> [1]PETSC ERROR: Libraries linked from
> /home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
> [1]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
> [1]PETSC ERROR: Configure options --download-mpich
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscMallocAlign() line 46 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
> src/AdLemMain on a arch-linux2-cxx-debug named edwards by bkhanal Thu Oct
> 17 15:19:22 2013
> [2]PETSC ERROR: Libraries linked from
> /home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
> [2]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
> [2]PETSC ERROR: Configure options --download-mpich
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: PetscMallocAlign() line 46 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
> [1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [1]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [2]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [2]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [1]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [2]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [2]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> [2]PETSC ERROR: [1]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [1]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [2]PETSC ERROR: DMCreateMatrix_DA() line 771 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> DMCreateMatrix_DA() line 771 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [3]PETSC ERROR: Memory allocated 0 Memory used by process 1675407360
> [3]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [3]PETSC ERROR: Memory requested 6166659200!
> [3]PETSC ERROR:
> ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [3]PETSC ERROR: See docs/changes/index.html for recent updates.
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
> [3]PETSC ERROR:
> ------------------------------------------------------------------------
> [3]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
> bkhanal Thu Oct 17 15:19:22 2013
> [3]PETSC ERROR: Libraries linked from
> /home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
> [3]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
> [3]PETSC ERROR: Configure options --download-mpich
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [3]PETSC ERROR:
> ------------------------------------------------------------------------
> [3]PETSC ERROR: [1]PETSC ERROR: DMCreateMatrix() line 910 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
> [2]PETSC ERROR: DMCreateMatrix() line 910 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
> PetscMallocAlign() line 46 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
> [3]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [3]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [1]PETSC ERROR: KSPSetUp() line 207 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: KSPSetUp() line 207 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [3]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [3]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [3]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [3]PETSC ERROR: DMCreateMatrix_DA() line 771 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [3]PETSC ERROR: DMCreateMatrix() line 910 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
> [3]PETSC ERROR: KSPSetUp() line 207 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: solveModel() line 128 in
> "unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> [2]PETSC ERROR: solveModel() line 128 in
> "unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> [3]PETSC ERROR: solveModel() line 128 in
> "unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> [0]PETSC ERROR: Memory allocated 0 Memory used by process 1711476736
> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [0]PETSC ERROR: Memory requested 6292477952!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: src/AdLemMain on a arch-linux2-cxx-debug named edwards by
> bkhanal Thu Oct 17 15:19:22 2013
> [0]PETSC ERROR: Libraries linked from
> /home/bkhanal/Documents/softwares/petsc-3.4.3/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Wed Oct 16 15:13:05 2013
> [0]PETSC ERROR: Configure options --download-mpich
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocAlign() line 46 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/sys/memory/mal.c
> [0]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3551 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: MatSeqAIJSetPreallocation() line 3496 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: MatMPIAIJSetPreallocation_MPIAIJ() line 3307 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: MatMPIAIJSetPreallocation() line 4015 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: DMCreateMatrix_DA_3d_MPIAIJ() line 1101 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: DMCreateMatrix_DA() line 771 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: DMCreateMatrix() line 910 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/dm/interface/dm.c
> [0]PETSC ERROR: KSPSetUp() line 207 in
> /home/bkhanal/Documents/softwares/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: solveModel() line 128 in
> "unknowndirectory/"/user/bkhanal/home/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> --9345:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
> --9345:0:aspacem    Increase it and rebuild.  Exiting now.
> --9344:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
> --9344:0:aspacem    Increase it and rebuild.  Exiting now.
> --9343:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
> --9343:0:aspacem    Increase it and rebuild.  Exiting now.
> --9346:0:aspacem  Valgrind: FATAL: VG_N_SEGMENTS is too low.
> --9346:0:aspacem    Increase it and rebuild.  Exiting now.
>
>
> ===================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   EXIT CODE: 1
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>
> ===================================================================================
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/883229c3/attachment-0001.html>

From jedbrown at mcs.anl.gov  Thu Oct 17 09:35:26 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 09:35:26 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
	<CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
Message-ID: <87li1sge35.fsf@mcs.anl.gov>

Bishesh Khanal <bisheshkh at gmail.com> writes:

> Yes, I tried it just a while ago and this is happened I think. (Just to
> confirm, I have put the error message for this case at the very end of this
> reply.*)

Yes, that is the "friendly" way to run out of memory.

> I'm sorry but I'm a complete beginner with MPI and clusters; so what does
> one MPI rank per node means and what should I do to do that ? My guess is
> that I set one core per node and use multiple nodes in my job script file ?
> Or do I need to do something in the petsc code ?

No, it is in the way you launch the job.  It's usually called "processes
per node", often with a key like "ppn", "nppn", or "mppnppn".  See the
documentation for the machine or ask your facility how to do it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/d87ab293/attachment.pgp>

From jedbrown at mcs.anl.gov  Thu Oct 17 09:39:20 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 09:39:20 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
	<CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
	<CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>
Message-ID: <87hacggdwn.fsf@mcs.anl.gov>

Bishesh Khanal <bisheshkh at gmail.com> writes:

>> I tried running on the cluster with one core per node with 4 nodes and I
> got the following errors (note: using valgrind, and openmpi of the cluster)
> at the very end after the many usual "unconditional jump ... errors"  which
> might be interesting
>
> mpiexec: killing job...
>
> mpiexec: abort is already in progress...hit ctrl-c again to forcibly
> terminate
>
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
> batch system) has told this process to end

Memory corruption generally results in SIGSEGV, so I suspect this is
still either a memory issue or some other resource issue.  How much
memory is available on these compute nodes?  Do turn off Valgrind for
this run; it takes a lot of memory.

> Does it mean it is crashing near MatSetValues_MPIAIJ ?

Possibly, but it could be killing the program for other reasons.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/d7c13a03/attachment.pgp>

From jedbrown at mcs.anl.gov  Thu Oct 17 09:50:36 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 09:50:36 -0500
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
Message-ID: <87eh7kgddv.fsf@mcs.anl.gov>

Shiva Rudraraju <rudraa at umich.edu> writes:

> I am planning to study the effects of geometric multigrid method both as a
> Solver and Preconditioner on a system of transport problems involving
> coupled diffusion + finite strain mechanics. Part of that study will be to
> evaluate the performance of FD vs higher order FEM (Both Lagrange and
> Spectral elements).

How do you distinguish "Lagrange" versus "Spectral" elements?  Are you
referring to nodal versus modal bases, choice of collocation quadrature,
or something else?

> So in this context I have a few questions:
> 1)  To motivate the use of MG schemes in this coupled system of equations,
> I plan to initially implement a Red-Black Gauss Siedel solver for FD to
> serve as a benchmark. Is there a good example related to the Red-Black
> Gauss Siedel implementation?.

You could reorder your equations, but multicolor GS is not a very good
or representative algorithm on cache-based architectures, due to its
poor cache reuse.  I suggest just using standard GS smoothers (-pc_type
sor with default relaxation parameter of 1.0).

> 2) I plan to implement a higher order Lagrange and Spectral element FEM
> code. I found one related example in
> petsc/src/ksp/ksp/examples/tutorials/ex59.c. Are there any better examples
> of general FEM code implementations.

Do you already have spectral element code or are you planning to write
one from scratch?  There are several unstructured FEM examples using
DMPlex, though they do not explicitly expose the tensor product that is
important for efficient high-order methods.  How would you plan to
precondition these things?

I did some work on this using embedded low-order discretizations (see
below) and the library used in that paper is available.  The paper used
algebraic multigrid, but geometric and semi-geometric methods would be
better in many settings.  (I'm hoping to merge the good attributes of
that library into PETSc using the new DMPlex.)

> 3) MG is often used as a preconditioner, but are there any examples showing
> its use as a solver?. 

That is just -ksp_type richardson.  Or are you talking about nonlinear
multigrid (-snes_type fas)?

> I found a few examples here
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/concepts/multigrid.html,
> but nothing in the current version of those codes refer to any MG
> datastructures like DMMG which were part of the earlier versions.

The DMMG object was a broken abstraction; its functionality was rolled into DM and SNES.



@article {brown2010ens,
  author = {Brown, Jed},
  affiliation = {Versuchsanstalt f?r Wasserbau, Hydrologie und Glaziologie (VAW), ETH Z?rich, Z?rich, Switzerland},
  title = {Efficient Nonlinear Solvers for Nodal High-Order Finite Elements in {3D}},
  journal = {Journal of Scientific Computing},
  publisher = {Springer Netherlands},
  issn = {0885-7474},
  keyword = {Mathematics and Statistics},
  pages = {48-63},
  volume = {45},
  issue = {1},
  url = {http://dx.doi.org/10.1007/s10915-010-9396-8},
  doi = {10.1007/s10915-010-9396-8},
  year = {2010}
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/00eed0cd/attachment.pgp>

From charlesj at purdue.edu  Thu Oct 17 10:02:45 2013
From: charlesj at purdue.edu (James A Charles)
Date: Thu, 17 Oct 2013 11:02:45 -0400 (EDT)
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <922743233.277989.1382021645592.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>

Hello,

I'm trying to use a Dense Parallel matrix where there are no nonzeros that is rectangular of size Nxp where N >> p. P is anywhere from 2-6 typically and N can be of size 1E6 or more. 

For this I would like to only distribute the rows along the MPI processes and not have the columns distributed at all (local columns size = global column size). What is the best way to do this in Petsc? If it matters I am using Petsc3.4. 

The operations performed after the Matrix allocation are:

direct call to Lapack for QR factorization via pointer to array. 

Matrix multiply with an NxN matrix. 

I have this working in serial but when I try to distribute my matrices I get columns with size p*(number of processes). 

Thanks,
James







From bsmith at mcs.anl.gov  Thu Oct 17 10:22:52 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 17 Oct 2013 10:22:52 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
References: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <3BBAC0D3-B46A-4FD7-9DB3-7BD65180E287@mcs.anl.gov>


  James,

   PETSc always distributes its MPIDense matrices this way.   The "local column" size doesn't mean which columns values are stored local. Yes the terminology we use is a bit confusing.  Just let PETSc pick the "number of local columns" in the matrix while you provide the global number of columns. In the actual storage of the dense matrix the "number of local columns" is ignored.

    Barry

On Oct 17, 2013, at 10:02 AM, James A Charles <charlesj at purdue.edu> wrote:

> Hello,
> 
> I'm trying to use a Dense Parallel matrix where there are no nonzeros that is rectangular of size Nxp where N >> p. P is anywhere from 2-6 typically and N can be of size 1E6 or more. 
> 
> For this I would like to only distribute the rows along the MPI processes and not have the columns distributed at all (local columns size = global column size). What is the best way to do this in Petsc? If it matters I am using Petsc3.4. 
> 
> The operations performed after the Matrix allocation are:
> 
> direct call to Lapack for QR factorization via pointer to array. 
> 
> Matrix multiply with an NxN matrix. 
> 
> I have this working in serial but when I try to distribute my matrices I get columns with size p*(number of processes). 
> 
> Thanks,
> James
> 
> 
> 
> 
> 
> 


From jedbrown at mcs.anl.gov  Thu Oct 17 10:29:01 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 10:29:01 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <3BBAC0D3-B46A-4FD7-9DB3-7BD65180E287@mcs.anl.gov>
References: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
	<3BBAC0D3-B46A-4FD7-9DB3-7BD65180E287@mcs.anl.gov>
Message-ID: <878uxrhq6a.fsf@mcs.anl.gov>

Barry Smith <bsmith at mcs.anl.gov> writes:
>    PETSc always distributes its MPIDense matrices this way.

See MATELEMENTAL if you want 2D distributions.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/122ddef8/attachment.pgp>

From bsmith at mcs.anl.gov  Thu Oct 17 10:30:18 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 17 Oct 2013 10:30:18 -0500
Subject: [petsc-users] signal received error; MatNullSpaceTest;
	Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
	<CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
	<CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>
Message-ID: <022A831C-9AC2-4943-B62C-8B254D07160B@mcs.anl.gov>


On Oct 17, 2013, at 9:26 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:
> 
> 
> --------------------------------------------------------------------------
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatSetValues_MPIAIJ line 505 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: [0] MatSetValues line 1071 /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] MatSetValuesLocal line 1935 /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] DMCreateMatrix_DA_3d_MPIAIJ line 1051 /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: [0] DMCreateMatrix_DA line 627 /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: [0] DMCreateMatrix line 900 /tmp/petsc-3.4.3/src/dm/interface/dm.c
> [0]PETSC ERROR: [0] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [0] solveModel line 122 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef002 by bkhanal Thu Oct 17 15:55:33 2013
> [0]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib
> [0]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1 --download-metis --download-parmetis --download-superlu_dist --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> ==47363== 
> ==47363== HEAP SUMMARY:
> ==47363==     in use at exit: 10,939,838,029 bytes in 8,091 blocks
> ==47363==   total heap usage: 1,936,963 allocs, 1,928,872 frees, 11,530,164,042 bytes allocated
> ==47363== 
> 
> Does it mean it is crashing near MatSetValues_MPIAIJ ?

> It is not really "crashing" here. The most likely cause is that the OS has run out of memory to provide so it has started shutting down processes; this is the "unfriendly" way that Unix handles "running out of memory".  A much less likely possibility (since the program has just started) is that the process has run out of allocated time and the OS is shutting it down.   

   I am confident in this case it is simply a mater of the system running out of memory.  What happens if you run the exact same job without the valgrind? Please send the ENTIRE error message.

   Barry





From bsmith at mcs.anl.gov  Thu Oct 17 10:31:06 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 17 Oct 2013 10:31:06 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <878uxrhq6a.fsf@mcs.anl.gov>
References: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
	<3BBAC0D3-B46A-4FD7-9DB3-7BD65180E287@mcs.anl.gov>
	<878uxrhq6a.fsf@mcs.anl.gov>
Message-ID: <CCF8B7CE-3FD7-42CB-AF2E-B6D47A7D1D37@mcs.anl.gov>


   He very clearly stated he did not want 2d distributions.

On Oct 17, 2013, at 10:29 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Barry Smith <bsmith at mcs.anl.gov> writes:
>>   PETSc always distributes its MPIDense matrices this way.
> 
> See MATELEMENTAL if you want 2D distributions.


From rudraa at umich.edu  Thu Oct 17 11:03:17 2013
From: rudraa at umich.edu (Shiva Rudraraju)
Date: Thu, 17 Oct 2013 12:03:17 -0400
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <87eh7kgddv.fsf@mcs.anl.gov>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
Message-ID: <CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>

Thanks Jed for the prompt reply.

> How do you distinguish "Lagrange" versus "Spectral" elements?  Are you referring
to nodal versus modal bases, choice of collocation quadrature, or something
else?
By Spectral Elements I mean spectral quadrilateral/hexahedral elements
based on tensor product lagrangian polynomials on Gauss Lobatto Legendre
points.

 >You could reorder your equations, but multicolor GS is not a very good or
representative algorithm on cache-based architectures, due to its poor
cache reuse.  I suggest just using standard GS smoothers (-pc_type sor with
default relaxation parameter of 1.0).
I plan to implement multicolor GS precisely to demonstrate its poor
performance as compared to other iterative and MG schemes, because in the
Phase Field community multicolor GS is still quite popular and lingers
around as a solver. The main point of this work is to clearly demonstrate
the ill-suitedness of GS for  these coupled transport problems.
So just wondering if there are any related examples showing multicolor GS
as a solver. Also, since you mentioned, are there any references which
demonstrate the poor cache reuse of multicolor GS or is it too obvious?...
just curious.

>Do you already have spectral element code or are you planning to write one
from scratch?  There are several unstructured FEM examples using DMPlex,
though they do not explicitly expose the tensor product that is
important for efficient high-order methods.  How would you plan to
precondition these things?

Sorry I forgot to mention..... I am only interested in structured quad/hex
elements. I have my old implementations of higher order Lagrange elements
and also used deal.ii's Spectral elements.... but for this work I will more
or less write one from scratch. So any pointers to efficient tensor grid
FEM implementation will really help me.

And thanks for your paper reference.

> That is just -ksp_type richardson.  Or are you talking about nonlinear
multigrid (-snes_type fas)?
Cool..... -snes_type fas is what I was looking for.

Thanks,
Shiva


On Thu, Oct 17, 2013 at 10:50 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Shiva Rudraraju <rudraa at umich.edu> writes:
>
> > I am planning to study the effects of geometric multigrid method both as
> a
> > Solver and Preconditioner on a system of transport problems involving
> > coupled diffusion + finite strain mechanics. Part of that study will be
> to
> > evaluate the performance of FD vs higher order FEM (Both Lagrange and
> > Spectral elements).
>
> How do you distinguish "Lagrange" versus "Spectral" elements?  Are you
> referring to nodal versus modal bases, choice of collocation quadrature,
> or something else?
>
> > So in this context I have a few questions:
> > 1)  To motivate the use of MG schemes in this coupled system of
> equations,
> > I plan to initially implement a Red-Black Gauss Siedel solver for FD to
> > serve as a benchmark. Is there a good example related to the Red-Black
> > Gauss Siedel implementation?.
>
> You could reorder your equations, but multicolor GS is not a very good
> or representative algorithm on cache-based architectures, due to its
> poor cache reuse.  I suggest just using standard GS smoothers (-pc_type
> sor with default relaxation parameter of 1.0).
>
> > 2) I plan to implement a higher order Lagrange and Spectral element FEM
> > code. I found one related example in
> > petsc/src/ksp/ksp/examples/tutorials/ex59.c. Are there any better
> examples
> > of general FEM code implementations.
>
> Do you already have spectral element code or are you planning to write
> one from scratch?  There are several unstructured FEM examples using
> DMPlex, though they do not explicitly expose the tensor product that is
> important for efficient high-order methods.  How would you plan to
> precondition these things?
>
> I did some work on this using embedded low-order discretizations (see
> below) and the library used in that paper is available.  The paper used
> algebraic multigrid, but geometric and semi-geometric methods would be
> better in many settings.  (I'm hoping to merge the good attributes of
> that library into PETSc using the new DMPlex.)
>
> > 3) MG is often used as a preconditioner, but are there any examples
> showing
> > its use as a solver?.
>
> That is just -ksp_type richardson.  Or are you talking about nonlinear
> multigrid (-snes_type fas)?
>
> > I found a few examples here
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/concepts/multigrid.html
> ,
> > but nothing in the current version of those codes refer to any MG
> > datastructures like DMMG which were part of the earlier versions.
>
> The DMMG object was a broken abstraction; its functionality was rolled
> into DM and SNES.
>
>
>
> @article {brown2010ens,
>   author = {Brown, Jed},
>   affiliation = {Versuchsanstalt f?r Wasserbau, Hydrologie und Glaziologie
> (VAW), ETH Z?rich, Z?rich, Switzerland},
>   title = {Efficient Nonlinear Solvers for Nodal High-Order Finite
> Elements in {3D}},
>   journal = {Journal of Scientific Computing},
>   publisher = {Springer Netherlands},
>   issn = {0885-7474},
>   keyword = {Mathematics and Statistics},
>   pages = {48-63},
>   volume = {45},
>   issue = {1},
>   url = {http://dx.doi.org/10.1007/s10915-010-9396-8},
>   doi = {10.1007/s10915-010-9396-8},
>   year = {2010}
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/9c5d5a4a/attachment.html>

From tsariysk at craft-tech.com  Thu Oct 17 12:10:32 2013
From: tsariysk at craft-tech.com (Ted Sariyski)
Date: Thu, 17 Oct 2013 13:10:32 -0400
Subject: [petsc-users] libpetsc.a(lgc.c.o): In function
 `PetscDrawLGDraw':lgc.c:469: undefined reference to `XSetIOErrorHandler'
In-Reply-To: <CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
Message-ID: <52601A08.4070203@craft-tech.com>

Hi,
I am trying to use ex10 (from snes/ex10d) and ex3p from MFEM (Finite 
Element Discretization Library), as starting templates.  Separately, I 
can work with both examples. Now, I am trying to merge ex10.c (PETSc) to 
ex3p.cpp (MFEM).  There are three lines changed in the original ex3p 
example: two include statements and replacement of MPI_Init with 
PetscInitialize:

#include "petscsnes.h"
#include "petscao.h"
...
// MPI_Init(&argc, &argv);
PetscInitialize(&argc,&argv,"options.inf",help);

At link time, I get a long list of "undefined reference to ..." (see 
below).

What do I miss here?
Thanks in advance,
--Ted

P.S.  The example from MFEM has a call to GLVis server. I wonder whether 
it may cause conflicts with PETSc visualization libraries?





ex10d: (works fine)
=================================
mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas 
-g   -o ex10 ../ex10.c -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
-I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
-I/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/include 
-I/usr/include -I/usr/include/openmpi-x86_64 
-L/home/tsariysk/MFEM/OMPI/PETSc_real/lib  -lpetsc -llapack -lblas -lX11 
-Wl,-rpath,/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetisusr 
-L/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetisusr 
-lmetis -lpthread -Wl,-rpath,/usr/lib64 -L/usr/lib64 -lhdf5 -lhdf5_cpp 
-lhdf5_fortran -lhdf5_hl -lhdf5_hl_cpp -lm -L/usr/lib64/openmpi/lib 
-L/opt/centos/devtoolset-1.1/root/usr/lib/gcc/x86_64-redhat-linux/4.7.2 
-L/opt/centos/devtoolset-1.1/root/usr/lib64 
-L/opt/centos/devtoolset-1.1/root/usr/lib -lmpi_f90 -lmpi_f77 -lgfortran 
-lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lgcc_s 
-lpthread -ldl
rm -rf ex10.o



ex3p: (eclipse Makefile)
=================================
g++ -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
-I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
-I/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/include 
-I/usr/include -I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib 
-L/home/tsariysk/MFEM/OMPI/PETSc_real/lib 
-L/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetis 
-L/home/tsariysk/MFEM/OMPI/hypre-2.9.0b/src/hypre/lib -g3 -o "SPHERE"  
./mesh/element.o ./mesh/hexahedron.o ./mesh/mesh.o ./mesh/mesquite.o 
./mesh/ncmesh.o ./mesh/nurbs.o ./mesh/pmesh.o ./mesh/point.o 
./mesh/quadrilateral.o ./mesh/segment.o ./mesh/tetrahedron.o 
./mesh/triangle.o ./mesh/vertex.o ./linalg/densemat.o ./linalg/hypre.o 
./linalg/matrix.o ./linalg/ode.o ./linalg/operator.o ./linalg/solvers.o 
./linalg/sparsemat.o ./linalg/sparsesmoothers.o ./linalg/vector.o 
./general/array.o ./general/communication.o ./general/error.o 
./general/isockstream.o ./general/osockstream.o ./general/sets.o 
./general/socketstream.o ./general/sort_pairs.o ./general/stable3d.o 
./general/table.o ./general/tic_toc.o  ./fem/bilinearform.o 
./fem/bilininteg.o ./fem/coefficient.o ./fem/eltrans.o ./fem/fe.o 
./fem/fe_coll.o ./fem/fespace.o ./fem/geom.o ./fem/gridfunc.o 
./fem/intrules.o ./fem/linearform.o ./fem/lininteg.o 
./fem/nonlinearform.o ./fem/nonlininteg.o ./fem/pbilinearform.o 
./fem/pfespace.o ./fem/pgridfunc.o ./fem/plinearform.o 
./fem/pnonlinearform.o  ./sphere.o   -lmpi_cxx -lmpi -ldl -lmetis 
-lHYPRE -lpetsc
/home/tsariysk/MFEM/OMPI/PETSc_real/lib/libpetsc.a(lgc.c.o): In function 
`PetscDrawLGDraw':
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/utils/lgc.c:469: 
undefined reference to `XSetIOErrorHandler'
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/utils/lgc.c:471: 
undefined reference to `XSetIOErrorHandler'
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/utils/lgc.c:519: 
undefined reference to `XSetIOErrorHandler'
/home/tsariysk/MFEM/OMPI/PETSc_real/lib/libpetsc.a(xops.c.o): In 
function `PetscDrawLine_X(_p_PetscDraw*, double, double, double, double, 
int)':
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xops.c:49: 
undefined reference to `XSetForeground'
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xops.c:53: 
undefined reference to `XDrawLine'
/home/tsariysk/MFEM/OMPI/PETSc_real/lib/libpetsc.a(xops.c.o): In 
function `PetscDrawArrow_X(_p_PetscDraw*, double, double, double, 
double, int)':
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xops.c:65: 
undefined reference to `XSetForeground'
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xops.c:69: 
undefined reference to `XDrawLine'
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xops.c:72: 
undefined reference to `XDrawLine'
...
...
/home/tsariysk/MFEM/petsc-3.4.1/src/sys/classes/draw/impls/x/xcolor.c:326: 
undefined reference to `XAllocColor'
collect2: error: ld returned 1 exit status


From knepley at gmail.com  Thu Oct 17 12:22:36 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 Oct 2013 12:22:36 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
References: <922743233.277989.1382021645592.JavaMail.root@mailhub050.itcs.purdue.edu>
	<320633801.278088.1382022165569.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <CAMYG4GmpzQj2Opn-kGB=drqViXwg80fNeuUH1rE4SOSxWU0GVg@mail.gmail.com>

On Thu, Oct 17, 2013 at 10:02 AM, James A Charles <charlesj at purdue.edu>wrote:

> Hello,
>
> I'm trying to use a Dense Parallel matrix where there are no nonzeros that
> is rectangular of size Nxp where N >> p. P is anywhere from 2-6 typically
> and N can be of size 1E6 or more.
>
> For this I would like to only distribute the rows along the MPI processes
> and not have the columns distributed at all (local columns size = global
> column size). What is the best way to do this in Petsc? If it matters I am
> using Petsc3.4.
>

We do not distribute columns. You could if you used the Elemental
implementation, but you do not need that.


> The operations performed after the Matrix allocation are:
>
> direct call to Lapack for QR factorization via pointer to array.
>

You really want Tall-Skinny QR here (TSQR). We have not implemented it, but
it is not hard, so if
you would like to contribute it, that would be great.


> Matrix multiply with an NxN matrix.
>

This works.

   Matt


> I have this working in serial but when I try to distribute my matrices I
> get columns with size p*(number of processes).
>
> Thanks,
> James
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/53f677d4/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 17 12:46:27 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 12:46:27 -0500
Subject: [petsc-users] libpetsc.a(lgc.c.o): In function
	`PetscDrawLGDraw':lgc.c:469: undefined reference to
	`XSetIOErrorHandler'
In-Reply-To: <52601A08.4070203@craft-tech.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
	<52601A08.4070203@craft-tech.com>
Message-ID: <87zjq7g58s.fsf@mcs.anl.gov>

When writing the list, please start new threads instead of replying to
old threads.

Ted Sariyski <tsariysk at craft-tech.com> writes:

> At link time, I get a long list of "undefined reference to ..." (see 
> below).
>
> What do I miss here?

When using static libraries, you have to link *all* the libraries:
-lpetsc is not sufficient.  You can use "make getlinklibs" to obtain the
full list.  Put that into Eclipse.

> ex10d: (works fine)
> =================================
> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas 
> -g   -o ex10 ../ex10.c -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
> -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
> -I/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/include 
> -I/usr/include -I/usr/include/openmpi-x86_64 
> -L/home/tsariysk/MFEM/OMPI/PETSc_real/lib  -lpetsc -llapack -lblas -lX11 
> -Wl,-rpath,/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetisusr 
> -L/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetisusr 
> -lmetis -lpthread -Wl,-rpath,/usr/lib64 -L/usr/lib64 -lhdf5 -lhdf5_cpp 
> -lhdf5_fortran -lhdf5_hl -lhdf5_hl_cpp -lm -L/usr/lib64/openmpi/lib 
> -L/opt/centos/devtoolset-1.1/root/usr/lib/gcc/x86_64-redhat-linux/4.7.2 
> -L/opt/centos/devtoolset-1.1/root/usr/lib64 
> -L/opt/centos/devtoolset-1.1/root/usr/lib -lmpi_f90 -lmpi_f77 -lgfortran 
> -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -ldl -lmpi -lgcc_s 
> -lpthread -ldl

You need all the libraries and paths above.

> ex3p: (eclipse Makefile)
> =================================
> g++ -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
> -I/home/tsariysk/MFEM/OMPI/PETSc_real/include 
> -I/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/include 
> -I/usr/include -I/usr/include/openmpi-x86_64 -L/usr/lib64/openmpi/lib 
> -L/home/tsariysk/MFEM/OMPI/PETSc_real/lib 
> -L/home/tsariysk/MFEM/OMPI/metis-5.1.0/build/Linux-x86_64/libmetis 
> -L/home/tsariysk/MFEM/OMPI/hypre-2.9.0b/src/hypre/lib -g3 -o "SPHERE"  
> ./mesh/element.o ./mesh/hexahedron.o ./mesh/mesh.o ./mesh/mesquite.o 
> ./mesh/ncmesh.o ./mesh/nurbs.o ./mesh/pmesh.o ./mesh/point.o 
> ./mesh/quadrilateral.o ./mesh/segment.o ./mesh/tetrahedron.o 
> ./mesh/triangle.o ./mesh/vertex.o ./linalg/densemat.o ./linalg/hypre.o 
> ./linalg/matrix.o ./linalg/ode.o ./linalg/operator.o ./linalg/solvers.o 
> ./linalg/sparsemat.o ./linalg/sparsesmoothers.o ./linalg/vector.o 
> ./general/array.o ./general/communication.o ./general/error.o 
> ./general/isockstream.o ./general/osockstream.o ./general/sets.o 
> ./general/socketstream.o ./general/sort_pairs.o ./general/stable3d.o 
> ./general/table.o ./general/tic_toc.o  ./fem/bilinearform.o 
> ./fem/bilininteg.o ./fem/coefficient.o ./fem/eltrans.o ./fem/fe.o 
> ./fem/fe_coll.o ./fem/fespace.o ./fem/geom.o ./fem/gridfunc.o 
> ./fem/intrules.o ./fem/linearform.o ./fem/lininteg.o 
> ./fem/nonlinearform.o ./fem/nonlininteg.o ./fem/pbilinearform.o 
> ./fem/pfespace.o ./fem/pgridfunc.o ./fem/plinearform.o 
> ./fem/pnonlinearform.o  ./sphere.o   -lmpi_cxx -lmpi -ldl -lmetis 
> -lHYPRE -lpetsc

This only does -lpetsc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/6d91e7ed/attachment.pgp>

From Andrew.Wilkins at csiro.au  Thu Oct 17 19:10:44 2013
From: Andrew.Wilkins at csiro.au (Andrew.Wilkins at csiro.au)
Date: Fri, 18 Oct 2013 00:10:44 +0000
Subject: [petsc-users] VISS and VIRS convergence problems
Message-ID: <57596E6532C9F54F9DF09BCDFE7B1D3A86466166@exmbx06-cdc.nexus.csiro.au>

Hi,

I'm new to the list.  I've been using PETSc as a linear solver for a few years, and now I'm starting to get into MOOSE which is a finite-element framework which uses PETSc much more optimally. 

I've got a problem where I'm using viss or virs (petsc3.3) to enforce a lower bound on my solution.  The actual application is unsaturated Darcy-Richards single-phase flow - slow water flow through a porous medium.  As water is withdrawn from a region, the pressure drops, but physically the water pressure is bounded from below.  The DE ensures this via a nonlinear relative permeability function which goes to zero at the lower-bound of pressure: permeability=0 means no water can flow from the region.  However, in the linearization, the nonlinear solver won't know about this function going to zero (unless it just happens to end up right at the lower bound), so often the pressure goes into the unphysical region (pressure < lower) during the solution process.  Therefore I'm using the VI solvers.

First question - the -snes_vi_type appears to make no difference at all.  What's it supposed to do?

Second question - So far, viss has converged always, but virs sometimes gives problems.  I've read in one email-thread that virs is preferred.  Why is that?  I'm basing my comment on only about 10 tests, but I'm still interested in your replies.  I'm pretty sure my jacobian is correct, as I have tested it using -snes_type=test, and also I have run using finite-differences, and still got the behaviour below.  This is the type of behaviour I get:

DT: 2.885464e+03
 Solving time step 48, time=406142.00...
  Initial |residual|_2 = 1.536061e-01
  NL step  0, |residual|_2 = 1.536061e-01
  0 SNES VI Function norm 1.536061428302e-01 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
    0 KSP Residual norm 1.536061428302e-01
    1 KSP Residual norm 5.954022644970e-05
    2 KSP Residual norm 1.874684877465e-07
  NL step  1, |residual|_2 = 2.508324e-02
  1 SNES VI Function norm 2.508323951297e-02 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
    0 KSP Residual norm 2.508323951297e-02
    1 KSP Residual norm 1.744363683367e-05
    2 KSP Residual norm 9.541678264164e-09
Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 1
Solve Did NOT Converge!
Solve failed... cutting timestep
DT: 1.442732e+03
 Solving time step 48, time=404699.00...
  Initial |residual|_2 = 1.536061e-01
  NL step  0, |residual|_2 = 1.536061e-01
  0 SNES VI Function norm 1.536061428302e-01 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
    0 KSP Residual norm 1.536061428302e-01
    1 KSP Residual norm 7.144446293019e-05
    2 KSP Residual norm 2.783195631751e-07
  NL step  1, |residual|_2 = 2.513316e-02
  1 SNES VI Function norm 2.513315575633e-02 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
    0 KSP Residual norm 2.513315575633e-02
    1 KSP Residual norm 2.187002762768e-05
    2 KSP Residual norm 1.996619992363e-08
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 1
Solve Did NOT Converge!
Solve failed... cutting timestep



Ph: +61 7 3327 4497.  Fax: +61 7 3327 4666
Queensland Centre for Advanced Technologies
PO Box 883, Kenmore, Qld, 4069  
 


From jedbrown at mcs.anl.gov  Thu Oct 17 21:36:03 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 17 Oct 2013 21:36:03 -0500
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
Message-ID: <87iowvfgq4.fsf@mcs.anl.gov>

Shiva Rudraraju <rudraa at umich.edu> writes:

> By Spectral Elements I mean spectral quadrilateral/hexahedral elements
> based on tensor product lagrangian polynomials on Gauss Lobatto Legendre
> points.

Okay "both Lagrange and Spectral elements" sounded like you wanted to
distinguish between two classes of methods.

>  >You could reorder your equations, but multicolor GS is not a very good or
> representative algorithm on cache-based architectures, due to its poor
> cache reuse.  I suggest just using standard GS smoothers (-pc_type sor with
> default relaxation parameter of 1.0).
> I plan to implement multicolor GS precisely to demonstrate its poor
> performance as compared to other iterative and MG schemes, because in the
> Phase Field community multicolor GS is still quite popular and lingers
> around as a solver. The main point of this work is to clearly demonstrate
> the ill-suitedness of GS for  these coupled transport problems.

Block Jacobi/SOR is still popular and useful.

> 
> So just wondering if there are any related examples showing multicolor
> GS as a solver. Also, since you mentioned, are there any references
> which demonstrate the poor cache reuse of multicolor GS or is it too
> obvious?...  just curious.

I though multicolor GS mostly died when cache-based architectures beat
out vector machines.  One well-optimized application that uses
multicolor GS is FUN3D, but it is doing nonlinear point-block
Gauss-Seidel with a second order residual and first-order correction,
and adds line smoothers for boundary layers.

> Sorry I forgot to mention..... I am only interested in structured quad/hex
> elements. I have my old implementations of higher order Lagrange elements
> and also used deal.ii's Spectral elements.... but for this work I will more
> or less write one from scratch. So any pointers to efficient tensor grid
> FEM implementation will really help me.

I did unstructured hexes.  You still haven't said what you'll use for
relaxation.  High-order discretizations tend to have poor h-ellipticity,
so they either need heavy smoothers or a correction based on a
discretization with better h-ellipticity.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131017/674687a4/attachment-0001.pgp>

From bisheshkh at gmail.com  Fri Oct 18 07:45:56 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 18 Oct 2013 14:45:56 +0200
Subject: [petsc-users] signal received error; MatNullSpaceTest;
 Stokes flow solver with pc fieldsplit and schur complement
In-Reply-To: <022A831C-9AC2-4943-B62C-8B254D07160B@mcs.anl.gov>
References: <CAEhex8j3S+FOXk=h_Fvgq=wXBZR4sVjuCi78PDxB0Zes=31cyw@mail.gmail.com>
	<CAJ98EDrOGtEbHzDBBrrY+NNdbN=_x7V+C4pdhJnHxHjT=eLqdQ@mail.gmail.com>
	<4C174936-B4FC-4B3C-B792-F3AC61436831@mcs.anl.gov>
	<44D5A873-DE81-43E5-872D-5B4E8B4F149D@lbl.gov>
	<CAEhex8hPpYmqLx22dfu7yQHcVtG-NQuNRQu=X-7x2BySBsTZhw@mail.gmail.com>
	<CAMYG4Gm8X=j9FcS1rbgPCoKNETOJEVHK7cVE47HbDW5NCvUP4w@mail.gmail.com>
	<alpine.LFD.2.03.1310161303190.9649@mcs.anl.gov>
	<CAEhex8j1km4nct3sHyZb5VyJXgsx0-dFw54uZoDDpX=SE84+5g@mail.gmail.com>
	<CAMYG4Gmi07qvtR+eNNncZXD9Z5kjFa7C-JOuVkThUuwKF2q3BQ@mail.gmail.com>
	<CAEhex8hGd1A7bcd41K7gfjWbzc-yLknNK3r18RN8BFh15oK38Q@mail.gmail.com>
	<87zjq8gigp.fsf@mcs.anl.gov>
	<CAEhex8hmfi-WOoeqXPmKGAKfmDmoOHDs5EFLrgVRFQ-cE5ZDCg@mail.gmail.com>
	<CAEhex8g5p++7CE_ai47OnmxQTvz5QL0as1inhQCevxCiEvX_Ng@mail.gmail.com>
	<022A831C-9AC2-4943-B62C-8B254D07160B@mcs.anl.gov>
Message-ID: <CAEhex8h8mx1q-5Wso+LOFbKCBdKF+u+e_v5w8Ha5K58RVf49uQ@mail.gmail.com>

On Thu, Oct 17, 2013 at 5:30 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Oct 17, 2013, at 9:26 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:
> >
> >
> >
> --------------------------------------------------------------------------
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
> batch system) has told this process to end
> > [0]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> > [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> > [0]PETSC ERROR: likely location of problem given in stack below
> > [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> > [0]PETSC ERROR:       INSTEAD the line number of the start of the
> function
> > [0]PETSC ERROR:       is given.
> > [0]PETSC ERROR: [0] MatSetValues_MPIAIJ line 505
> /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: [0] MatSetValues line 1071
> /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: [0] MatSetValuesLocal line 1935
> /tmp/petsc-3.4.3/src/mat/interface/matrix.c
> > [0]PETSC ERROR: [0] DMCreateMatrix_DA_3d_MPIAIJ line 1051
> /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
> > [0]PETSC ERROR: [0] DMCreateMatrix_DA line 627
> /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c
> > [0]PETSC ERROR: [0] DMCreateMatrix line 900
> /tmp/petsc-3.4.3/src/dm/interface/dm.c
> > [0]PETSC ERROR: [0] KSPSetUp line 192
> /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: [0] solveModel line 122
> "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Signal received!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR:
> /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a
> arch-linux2-cxx-debug named nef002 by bkhanal Thu Oct 17 15:55:33 2013
> > [0]PETSC ERROR: Libraries linked from
> /epi/asclepios2/bkhanal/petscDebug/lib
> > [0]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013
> > [0]PETSC ERROR: Configure options
> --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries
> --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1
> --download-metis --download-parmetis --download-superlu_dist
> --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> > ==47363==
> > ==47363== HEAP SUMMARY:
> > ==47363==     in use at exit: 10,939,838,029 bytes in 8,091 blocks
> > ==47363==   total heap usage: 1,936,963 allocs, 1,928,872 frees,
> 11,530,164,042 bytes allocated
> > ==47363==
> >
> > Does it mean it is crashing near MatSetValues_MPIAIJ ?
>
> > It is not really "crashing" here. The most likely cause is that the OS
> has run out of memory to provide so it has started shutting down processes;
> this is the "unfriendly" way that Unix handles "running out of memory".  A
> much less likely possibility (since the program has just started) is that
> the process has run out of allocated time and the OS is shutting it down.
>
>    I am confident in this case it is simply a mater of the system running
> out of memory.  What happens if you run the exact same job without the
> valgrind? Please send the ENTIRE error message.
>

Thanks!! Yes it seems that it was because of the running out of the memory.
I realized that in the job script I submitted to the cluster I had limited
the memory to a certain value. I deleted that line and resubmitted the job,
now I the job gets killed because of the wall time (of 3 hours) being
exceeded unlike the previous case where the mpiexec killed the job quite
quickly. Now I will run the job again by increasing the wall time and see
if it gives results or not and get back here. Thanks again!


>
>    Barry
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/ee2b1734/attachment.html>

From einar.sorheim at gmail.com  Fri Oct 18 08:30:54 2013
From: einar.sorheim at gmail.com (=?ISO-8859-1?Q?Einar_S=F8rheim?=)
Date: Fri, 18 Oct 2013 15:30:54 +0200
Subject: [petsc-users] Problem with GAMG elasticity
 case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <87txggipud.fsf@mcs.anl.gov>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
Message-ID: <CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>

Uncommenting  KSPSetFromOptions solved my intial problem, however I got a
new problem that occurs after many timesteps, in our code we use petsc gamg
as a solver for several problems(e.g. thermal and mechanical) at each time
step, I have a suspicion it might have something to do with cleanup after
solving, the rror appears after many calls to the petsc gamg solver
interface, the error message is the following:

[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR:
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatSetNearNullSpace line 7580
src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by
admeinar Fri Oct 18 12:26:04 2013
[0]PETSC ERROR: Libraries linked from
/cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
[0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
[0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes
--with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
--with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
(x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
--with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
(x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0

The source code for the petsc interface:

      subroutine
petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond,
i_Solver
     $     ,coords_,i_nodedof)
#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscpc.h>
#include <finclude/petscksp.h>
#include <finclude/petscviewer.h>

      integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
      integer i_rowstart_(*),i_columnindex_(*)
      double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
      integer, allocatable  :: i_indices_(:)
!     Variables:
!
!  A       - matrix that defines linear system
!  ksp    - KSP context
!  ksp     - KSP context
!  x, b, u - approx solution, RHS, exact solution vectors
!
!      implicit none
      Vec     x,u,b,vec_coords
      PC      pc
      Mat     A,F,Matnull
      KSP    ksp
      PetscInt i,j,II,JJ,m,n,i1,imaxiter
      PetscInt Istart,Iend
      PetscInt nsteps,one, neg_one
      PetscErrorCode ierr
      PetscBool  flg
      PetscScalar  v,x_array_(1), norm, tol, t0,t1
      PetscOffset i_x
      PetscViewer viewer


      call PetscTime(t0,ierr)
      write(*,*) "Enter petsc"

      n      = 1
      nsteps = 1
      one    = 1
      neg_one = 1
      i1 = 1
      imaxiter = 1200


      call MatCreate(PETSC_COMM_WORLD,A,ierr)
      call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
      call MatSetType(A, MATSEQAIJ,ierr)
      call MatSetUp(A,ierr)
      call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
!  The matrix is partitioned by contiguous chunks of rows across the
!  processors.  Determine which rows of the matrix are locally owned.

      call MatGetOwnershipRange(A,Istart,Iend,ierr)


      write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
      do i=1,i_nodes
!         write(*,*) "Start of loop",i !!, i_indices_(i+1)
!         i_indices_(i+1) = i
!         write(*,*) "Start of loop2"
         do j=i_rowstart_(i),i_rowstart_(i+1)-1
!            write(*,*) "Start of loop 3"
            v = r_elements_(j)
!            write(*,*) i1,i,i1,j-1,v
            call
MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
            if (ierr.gt.0) stop
         end do
      end do
!      write(*,*) "End of matrix fill"

!  Assemble matrix, using the 2-step process:
!       MatAssemblyBegin(), MatAssemblyEnd()
!  Computations can be done while messages are in transition
!  by placing code between these two statements.

      call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)

!  Create parallel vectors.
!   - When using VecCreate(), the parallel partitioning of the vector
!     is determined by PETSc at runtime.
!   - Note: We form 1 vector from scratch and then duplicate as needed.
!      write(*,*) "Start of vector fill"
      call VecCreate(PETSC_COMM_WORLD,u,ierr)
      call VecSetType(u, VECSEQ,ierr)
      call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
!      call VecSetFromOptions(u,ierr)
      call VecDuplicate(u,b,ierr)
      call VecDuplicate(b,x,ierr)
      do i=1,i_nodes
         call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
         call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
      enddo
!  Assemble vector

       call VecAssemblyBegin(x,ierr)
       call VecAssemblyEnd(x,ierr)
       call VecAssemblyBegin(b,ierr)
       call VecAssemblyEnd(b,ierr)
       call PetscTime(t1,ierr)
!       write(*,*) "End of vec fill time spent :", t1-t0
!  Create linear solver context

      call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)


!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.

      call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)


      call KSPGetPC(ksp,pc,ierr)
      tol = 1.e-20
      call
KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
      select case (i_Precond)
      case ( 1 )
         call PCSetType(pc,PCJACOBI,ierr)
      case ( 2 )
         call PCSetType(pc,PCILU,ierr)
         call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
         call PCFactorSetUpMatSolverPackage(pc,ierr);
         call PCFactorGetMatrix(pc,F);
         call MatSuperluSetILUDropTol(F,1.e-4);
      case ( 3 )
         call PCSetType(pc,PCGAMG,ierr)
!         call PCGAMGInitializePackage()
!         call PCCreate_GAMG(pc,ierr)
      case DEFAULT
         call PCSetType(pc,PCJACOBI,ierr)
      end select

!      call PCSetType(pc,PCJACOBI,ierr)
      select case (i_solver)
      case ( 0 )
         call KSPSetType(ksp,KSPCG,ierr)
      case DEFAULT
         call KSPSetType(ksp,KSPCG,ierr)
      end select
      call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
      if (i_nodedof==3) then
         write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
         call
VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
         call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
         call MatSetNearNullSpace(A,Matnull,ierr)
         call MatNullSpaceDestroy(Matnull,ierr)

         write(*,*) "Finish setting null space ierr =", ierr
!         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )

      end if
!      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL,
PETSC_NULL,ierr)
!  Set runtime options, e.g.,
!      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
!  These options will override those specified above as long as
!  KSPSetFromOptions() is called _after_ any other customization
!  routines.

      call KSPSetFromOptions(ksp,ierr)
      call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
      call KSPSetUp(ksp,ierr)
      call PetscTime(t0,ierr)
      write(*,*) "Time setup", t0-t1
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      call KSPSolve(ksp,b,x,ierr)
      call PetscTime(t1,ierr)
      write(*,*) "end solve, time used:",t1-t0
      call KSPGetSolution(ksp,x,ierr)
      call VecGetArray(x,x_array_,i_x,ierr)
      do i=1,i_nodes
         r_x_(i) = x_array_(i_x + i)
      enddo
      call VecRestoreArray(x,x_array_,i_x,ierr)


!  Free work space.  All PETSc objects should be destroyed when they
!  are no longer needed.
!      call PCDestroy(pc,ierr)

      call VecDestroy(u,ierr)
      call VecDestroy(x,ierr)
      call VecDestroy(b,ierr)
      call MatDestroy(A,ierr)
      if (i_nodedof==3) then
         call VecDestroy(vec_coords,ierr)
         call MatDestroy(Matnull,ierr)
      endif
      call KSPDestroy(ksp,ierr)
      call PetscTime(t0,ierr)
      write(*,*) "time celan up :", t0-t1

      end

-- 
Einar S?rheim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/e7d1d8b2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 851 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/e7d1d8b2/attachment-0001.pgp>

From mfadams at lbl.gov  Fri Oct 18 09:08:26 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Fri, 18 Oct 2013 10:08:26 -0400
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
Message-ID: <6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>

I would guess there is some memory corruption or leak someplace.  Try running with -on_error_attach_debugger, and try to get some more information.  Do you just repeat a call to the code below many times?  Is this deterministic, does it fail on the same solve each time? 


On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:

> Uncommenting  KSPSetFromOptions solved my intial problem, however I got a new problem that occurs after many timesteps, in our code we use petsc gamg as a solver for several problems(e.g. thermal and mechanical) at each time step, I have a suspicion it might have something to do with cleanup after solving, the rror appears after many calls to the petsc gamg solver interface, the error message is the following:
> 
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580 src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Fri Oct 18 12:26:04 2013
> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> 
> The source code for the petsc interface: 
> 
>       subroutine petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond, i_Solver
>      $     ,coords_,i_nodedof)
> #include <finclude/petscsys.h>
> #include <finclude/petscvec.h>
> #include <finclude/petscmat.h>
> #include <finclude/petscpc.h>
> #include <finclude/petscksp.h>
> #include <finclude/petscviewer.h>
> 
>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
>       integer i_rowstart_(*),i_columnindex_(*)
>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
>       integer, allocatable  :: i_indices_(:)
> !     Variables:
> !
> !  A       - matrix that defines linear system
> !  ksp    - KSP context
> !  ksp     - KSP context
> !  x, b, u - approx solution, RHS, exact solution vectors
> !
> !      implicit none
>       Vec     x,u,b,vec_coords
>       PC      pc
>       Mat     A,F,Matnull
>       KSP    ksp
>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
>       PetscInt Istart,Iend
>       PetscInt nsteps,one, neg_one
>       PetscErrorCode ierr
>       PetscBool  flg
>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
>       PetscOffset i_x
>       PetscViewer viewer
> 
> 
>       call PetscTime(t0,ierr)
>       write(*,*) "Enter petsc"
> 
>       n      = 1
>       nsteps = 1
>       one    = 1
>       neg_one = 1
>       i1 = 1
>       imaxiter = 1200
> 
> 
>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
>       call MatSetType(A, MATSEQAIJ,ierr)
>       call MatSetUp(A,ierr)
>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
> !  The matrix is partitioned by contiguous chunks of rows across the
> !  processors.  Determine which rows of the matrix are locally owned. 
> 
>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
> 
> 
>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
>       do i=1,i_nodes
> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
> !         i_indices_(i+1) = i
> !         write(*,*) "Start of loop2"
>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
> !            write(*,*) "Start of loop 3"
>             v = r_elements_(j)
> !            write(*,*) i1,i,i1,j-1,v
>             call MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
>             if (ierr.gt.0) stop
>          end do
>       end do
> !      write(*,*) "End of matrix fill"
> 
> !  Assemble matrix, using the 2-step process:
> !       MatAssemblyBegin(), MatAssemblyEnd()
> !  Computations can be done while messages are in transition
> !  by placing code between these two statements.
> 
>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
> 
> !  Create parallel vectors.
> !   - When using VecCreate(), the parallel partitioning of the vector
> !     is determined by PETSc at runtime.
> !   - Note: We form 1 vector from scratch and then duplicate as needed.
> !      write(*,*) "Start of vector fill"
>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
>       call VecSetType(u, VECSEQ,ierr)
>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
> !      call VecSetFromOptions(u,ierr)      
>       call VecDuplicate(u,b,ierr)
>       call VecDuplicate(b,x,ierr)
>       do i=1,i_nodes
>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
>       enddo
> !  Assemble vector
> 
>        call VecAssemblyBegin(x,ierr)
>        call VecAssemblyEnd(x,ierr)
>        call VecAssemblyBegin(b,ierr)
>        call VecAssemblyEnd(b,ierr)
>        call PetscTime(t1,ierr)
> !       write(*,*) "End of vec fill time spent :", t1-t0
> !  Create linear solver context
> 
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> 
> 
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
> 
>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
> 
> 
>       call KSPGetPC(ksp,pc,ierr)
>       tol = 1.e-20
>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>       select case (i_Precond)
>       case ( 1 )
>          call PCSetType(pc,PCJACOBI,ierr)
>       case ( 2 )
>          call PCSetType(pc,PCILU,ierr)
>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
>          call PCFactorSetUpMatSolverPackage(pc,ierr); 
>          call PCFactorGetMatrix(pc,F);
>          call MatSuperluSetILUDropTol(F,1.e-4);
>       case ( 3 )
>          call PCSetType(pc,PCGAMG,ierr)
> !         call PCGAMGInitializePackage()
> !         call PCCreate_GAMG(pc,ierr)
>       case DEFAULT
>          call PCSetType(pc,PCJACOBI,ierr)
>       end select
> 
> !      call PCSetType(pc,PCJACOBI,ierr)
>       select case (i_solver)
>       case ( 0 )
>          call KSPSetType(ksp,KSPCG,ierr)
>       case DEFAULT
>          call KSPSetType(ksp,KSPCG,ierr)
>       end select
>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>       if (i_nodedof==3) then
>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>          call MatSetNearNullSpace(A,Matnull,ierr)
>          call MatNullSpaceDestroy(Matnull,ierr)
> 
>          write(*,*) "Finish setting null space ierr =", ierr
> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
> 
>       end if
> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL, PETSC_NULL,ierr)
> !  Set runtime options, e.g.,
> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
> !  These options will override those specified above as long as
> !  KSPSetFromOptions() is called _after_ any other customization
> !  routines.
> 
>       call KSPSetFromOptions(ksp,ierr)
>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>       call KSPSetUp(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "Time setup", t0-t1
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
> 
>       call KSPSolve(ksp,b,x,ierr)
>       call PetscTime(t1,ierr)
>       write(*,*) "end solve, time used:",t1-t0
>       call KSPGetSolution(ksp,x,ierr)
>       call VecGetArray(x,x_array_,i_x,ierr)
>       do i=1,i_nodes
>          r_x_(i) = x_array_(i_x + i)
>       enddo
>       call VecRestoreArray(x,x_array_,i_x,ierr)
> 
> 
> !  Free work space.  All PETSc objects should be destroyed when they
> !  are no longer needed.
> !      call PCDestroy(pc,ierr)
> 
>       call VecDestroy(u,ierr)
>       call VecDestroy(x,ierr)
>       call VecDestroy(b,ierr)
>       call MatDestroy(A,ierr)
>       if (i_nodedof==3) then
>          call VecDestroy(vec_coords,ierr)
>          call MatDestroy(Matnull,ierr)
>       endif
>       call KSPDestroy(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "time celan up :", t0-t1
> 
>       end
> 
> -- 
> Einar S?rheim
> <Mail Attachment>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/a24aca8f/attachment.html>

From torquil at gmail.com  Fri Oct 18 09:42:19 2013
From: torquil at gmail.com (=?UTF-8?Q?Torquil_Macdonald_S=C3=B8rensen?=)
Date: Fri, 18 Oct 2013 16:42:19 +0200
Subject: [petsc-users] CPU-scaling
Message-ID: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>

Hi!

I'm just starting to use SLEPc on a quad-core workstation computer (Intel
Core i5-2500 CPU @ 3.30GHz).

But I'm not getting the speedup that I was expecting, when increasing the
number of cores. E.g. when running ex1.c from the documentation, on 1, 2, 3
and 4 cores. When asking for the 500 eigenvalues of smallest magnitude, for
a 5000x5000 matrix, I'm getting the following approximate timings:

# n_core  time_usage
1              480s
2              400s
3              500s
4              580s

Should I not expect a speedup for n_core > 2?

The example is using the krylovschur algorithm. For all four runs, the
program states "Number of iterations of the method: 33".

I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is
compiled using GCC 4.8.1 or 4.8.2.

Best regards
Torquil S?rensen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/5124e4c6/attachment-0001.html>

From jroman at dsic.upv.es  Fri Oct 18 09:46:04 2013
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 18 Oct 2013 16:46:04 +0200
Subject: [petsc-users] CPU-scaling
In-Reply-To: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
References: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
Message-ID: <0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>


El 18/10/2013, a las 16:42, Torquil Macdonald S?rensen escribi?:

> Hi!
> 
> I'm just starting to use SLEPc on a quad-core workstation computer (Intel Core i5-2500 CPU @ 3.30GHz).
> 
> But I'm not getting the speedup that I was expecting, when increasing the number of cores. E.g. when running ex1.c from the documentation, on 1, 2, 3 and 4 cores. When asking for the 500 eigenvalues of smallest magnitude, for a 5000x5000 matrix, I'm getting the following approximate timings:
> 
> # n_core  time_usage
> 1              480s
> 2              400s
> 3              500s
> 4              580s
> 
> Should I not expect a speedup for n_core > 2?
> 
> The example is using the krylovschur algorithm. For all four runs, the program states "Number of iterations of the method: 33".
> 
> I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is compiled using GCC 4.8.1 or 4.8.2.
> 
> Best regards
> Torquil S?rensen
> 

Is it a generalized eigenproblem and/or are you doing shift-and-invert? In that case, have a look at FAQ #6 http://www.grycap.upv.es/slepc/documentation/faq.htm

Jose


From rudraa at umich.edu  Fri Oct 18 10:15:04 2013
From: rudraa at umich.edu (Shiva Rudraraju)
Date: Fri, 18 Oct 2013 11:15:04 -0400
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <87iowvfgq4.fsf@mcs.anl.gov>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
	<87iowvfgq4.fsf@mcs.anl.gov>
Message-ID: <CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>

>I did unstructured hexes.  You still haven't said what you'll use for relaxation.
 High-order discretizations tend to have poor h-ellipticity, so they either
need heavy smoothers or a correction based on a discretization with better
h-ellipticity.
Quite frankly, I was not aware of the poor h-ellipticity of higher order
elements and I was assuming I would use the regular GS/GMRES/etc for
relaxation. I looked up h-ellipticity of higher order elements and now this
adds to my worries :(. I may be asking for too much here.... but what do
you mean by heavy smoothers? or correction based on a discretization?.


On Thu, Oct 17, 2013 at 10:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Shiva Rudraraju <rudraa at umich.edu> writes:
>
> > By Spectral Elements I mean spectral quadrilateral/hexahedral elements
> > based on tensor product lagrangian polynomials on Gauss Lobatto Legendre
> > points.
>
> Okay "both Lagrange and Spectral elements" sounded like you wanted to
> distinguish between two classes of methods.
>
> >  >You could reorder your equations, but multicolor GS is not a very good
> or
> > representative algorithm on cache-based architectures, due to its poor
> > cache reuse.  I suggest just using standard GS smoothers (-pc_type sor
> with
> > default relaxation parameter of 1.0).
> > I plan to implement multicolor GS precisely to demonstrate its poor
> > performance as compared to other iterative and MG schemes, because in the
> > Phase Field community multicolor GS is still quite popular and lingers
> > around as a solver. The main point of this work is to clearly demonstrate
> > the ill-suitedness of GS for  these coupled transport problems.
>
> Block Jacobi/SOR is still popular and useful.
>
> >
> > So just wondering if there are any related examples showing multicolor
> > GS as a solver. Also, since you mentioned, are there any references
> > which demonstrate the poor cache reuse of multicolor GS or is it too
> > obvious?...  just curious.
>
> I though multicolor GS mostly died when cache-based architectures beat
> out vector machines.  One well-optimized application that uses
> multicolor GS is FUN3D, but it is doing nonlinear point-block
> Gauss-Seidel with a second order residual and first-order correction,
> and adds line smoothers for boundary layers.
>
> > Sorry I forgot to mention..... I am only interested in structured
> quad/hex
> > elements. I have my old implementations of higher order Lagrange elements
> > and also used deal.ii's Spectral elements.... but for this work I will
> more
> > or less write one from scratch. So any pointers to efficient tensor grid
> > FEM implementation will really help me.
>
> I did unstructured hexes.  You still haven't said what you'll use for
> relaxation.  High-order discretizations tend to have poor h-ellipticity,
> so they either need heavy smoothers or a correction based on a
> discretization with better h-ellipticity.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/b89ca21c/attachment.html>

From knepley at gmail.com  Fri Oct 18 10:21:34 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 Oct 2013 10:21:34 -0500
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
	<87iowvfgq4.fsf@mcs.anl.gov>
	<CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>
Message-ID: <CAMYG4Gkdx1wdw1Wyx0EKBzStk9k8M8HPZEuiE9VzW4oT+tztiw@mail.gmail.com>

On Fri, Oct 18, 2013 at 10:15 AM, Shiva Rudraraju <rudraa at umich.edu> wrote:

> >I did unstructured hexes.  You still haven't said what you'll use for relaxation.
>  High-order discretizations tend to have poor h-ellipticity, so they
> either need heavy smoothers or a correction based on a discretization
> with better h-ellipticity.
> Quite frankly, I was not aware of the poor h-ellipticity of higher order
> elements and I was assuming I would use the regular GS/GMRES/etc for
> relaxation. I looked up h-ellipticity of higher order elements and now this
> adds to my worries :(. I may be asking for too much here.... but what do
> you mean by heavy smoothers? or correction based on a discretization?.
>

By "heavy" Jed means a powerful but expensive solver. By "correction based
on a discretization" Jed means
a smoother which is tailored to the specific higher order discretization
that probably will not work in the
general case (and would need to be coded up).

    Matt


> On Thu, Oct 17, 2013 at 10:36 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Shiva Rudraraju <rudraa at umich.edu> writes:
>>
>> > By Spectral Elements I mean spectral quadrilateral/hexahedral elements
>> > based on tensor product lagrangian polynomials on Gauss Lobatto Legendre
>> > points.
>>
>> Okay "both Lagrange and Spectral elements" sounded like you wanted to
>> distinguish between two classes of methods.
>>
>> >  >You could reorder your equations, but multicolor GS is not a very
>> good or
>> > representative algorithm on cache-based architectures, due to its poor
>> > cache reuse.  I suggest just using standard GS smoothers (-pc_type sor
>> with
>> > default relaxation parameter of 1.0).
>> > I plan to implement multicolor GS precisely to demonstrate its poor
>> > performance as compared to other iterative and MG schemes, because in
>> the
>> > Phase Field community multicolor GS is still quite popular and lingers
>> > around as a solver. The main point of this work is to clearly
>> demonstrate
>> > the ill-suitedness of GS for  these coupled transport problems.
>>
>> Block Jacobi/SOR is still popular and useful.
>>
>> >
>> > So just wondering if there are any related examples showing multicolor
>> > GS as a solver. Also, since you mentioned, are there any references
>> > which demonstrate the poor cache reuse of multicolor GS or is it too
>> > obvious?...  just curious.
>>
>> I though multicolor GS mostly died when cache-based architectures beat
>> out vector machines.  One well-optimized application that uses
>> multicolor GS is FUN3D, but it is doing nonlinear point-block
>> Gauss-Seidel with a second order residual and first-order correction,
>> and adds line smoothers for boundary layers.
>>
>> > Sorry I forgot to mention..... I am only interested in structured
>> quad/hex
>> > elements. I have my old implementations of higher order Lagrange
>> elements
>> > and also used deal.ii's Spectral elements.... but for this work I will
>> more
>> > or less write one from scratch. So any pointers to efficient tensor grid
>> > FEM implementation will really help me.
>>
>> I did unstructured hexes.  You still haven't said what you'll use for
>> relaxation.  High-order discretizations tend to have poor h-ellipticity,
>> so they either need heavy smoothers or a correction based on a
>> discretization with better h-ellipticity.
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/d151d217/attachment.html>

From torquil at gmail.com  Fri Oct 18 10:21:32 2013
From: torquil at gmail.com (=?UTF-8?Q?Torquil_Macdonald_S=C3=B8rensen?=)
Date: Fri, 18 Oct 2013 17:21:32 +0200
Subject: [petsc-users] CPU-scaling
In-Reply-To: <0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>
References: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
	<0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>
Message-ID: <CAD6PAKMWR9SqHEicN_k=KWgnoQmftUNgT+SRZa6TYig8zx3YhQ@mail.gmail.com>

It is a standard eigenvalue problem, and I'm only using default algorithm
options, so I don't think I'm using any nontrivial spectral transformation.

The program is just ex1.c from the documentation, available here:

http://www.grycap.upv.es/slepc/documentation/current/src/eps/examples/tutorials/ex1.c

FAQ #6 mentions MUMPS, which I don't have the moment, so I'll have to look
into that.

- Torquil



On 18 October 2013 16:46, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 18/10/2013, a las 16:42, Torquil Macdonald S?rensen escribi?:
>
> > Hi!
> >
> > I'm just starting to use SLEPc on a quad-core workstation computer
> (Intel Core i5-2500 CPU @ 3.30GHz).
> >
> > But I'm not getting the speedup that I was expecting, when increasing
> the number of cores. E.g. when running ex1.c from the documentation, on 1,
> 2, 3 and 4 cores. When asking for the 500 eigenvalues of smallest
> magnitude, for a 5000x5000 matrix, I'm getting the following approximate
> timings:
> >
> > # n_core  time_usage
> > 1              480s
> > 2              400s
> > 3              500s
> > 4              580s
> >
> > Should I not expect a speedup for n_core > 2?
> >
> > The example is using the krylovschur algorithm. For all four runs, the
> program states "Number of iterations of the method: 33".
> >
> > I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is
> compiled using GCC 4.8.1 or 4.8.2.
> >
> > Best regards
> > Torquil S?rensen
> >
>
> Is it a generalized eigenproblem and/or are you doing shift-and-invert? In
> that case, have a look at FAQ #6
> http://www.grycap.upv.es/slepc/documentation/faq.htm
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/5a5955c6/attachment-0001.html>

From jedbrown at mcs.anl.gov  Fri Oct 18 10:28:42 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 Oct 2013 10:28:42 -0500
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
	<87iowvfgq4.fsf@mcs.anl.gov>
	<CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>
Message-ID: <87sivyegyd.fsf@mcs.anl.gov>

Shiva Rudraraju <rudraa at umich.edu> writes:

>>I did unstructured hexes.  You still haven't said what you'll use for relaxation.
>  High-order discretizations tend to have poor h-ellipticity, so they either
> need heavy smoothers or a correction based on a discretization with better
> h-ellipticity.
> Quite frankly, I was not aware of the poor h-ellipticity of higher order
> elements and I was assuming I would use the regular GS/GMRES/etc for
> relaxation. I looked up h-ellipticity of higher order elements and now this
> adds to my worries :(. I may be asking for too much here.... but what do
> you mean by heavy smoothers? or correction based on a discretization?.

You can use one discretization for evaluating residuals, but then use an
embedded low-order discretization to apply the correction.  See "defect
correction" in Achi's multigrid guide or in Trottenberg.

The paper I mentioned earlier was doing something simpler and less
intrusive: assemble the embedded low-order operator and feed it to
algebraic multigrid, but evaluate residuals matrix-free using the
high-order discretization.  But if you have a reasonable geometric
hierarchy, the defect correction schemes are better.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/08523986/attachment.pgp>

From rudraa at umich.edu  Fri Oct 18 11:04:28 2013
From: rudraa at umich.edu (Shiva Rudraraju)
Date: Fri, 18 Oct 2013 12:04:28 -0400
Subject: [petsc-users] Geometric MG as Solver & Preconditioner for
	FEM/Spectral/FD
In-Reply-To: <87sivyegyd.fsf@mcs.anl.gov>
References: <CADuYZoiLr_86R+QuVDy=z4R+d+TC_ZHDmDkPtjUgsUGCn_Xm6Q@mail.gmail.com>
	<87eh7kgddv.fsf@mcs.anl.gov>
	<CADuYZogWsvNjTU=5BiNpaF9r1w_bcGEOSDnuOQTnPoG65OKrsQ@mail.gmail.com>
	<87iowvfgq4.fsf@mcs.anl.gov>
	<CADuYZoi4qiiS17QiDbB0COD60Dd6EfZbu94iheSC8V7YD8gkpQ@mail.gmail.com>
	<87sivyegyd.fsf@mcs.anl.gov>
Message-ID: <CADuYZoix9hAPscqO66LppUgWCxOZzwtZS47nZXMGmU0Jf8y5WA@mail.gmail.com>

> You can use one discretization for evaluating residuals, but then use an
embedded low-order discretization to apply the correction.  See "defect
correction" in Achi's multigrid guide or in Trottenberg.
Thanks Jed and Matt. Will check out these references.


On Fri, Oct 18, 2013 at 11:28 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Shiva Rudraraju <rudraa at umich.edu> writes:
>
> >>I did unstructured hexes.  You still haven't said what you'll use for
> relaxation.
> >  High-order discretizations tend to have poor h-ellipticity, so they
> either
> > need heavy smoothers or a correction based on a discretization with
> better
> > h-ellipticity.
> > Quite frankly, I was not aware of the poor h-ellipticity of higher order
> > elements and I was assuming I would use the regular GS/GMRES/etc for
> > relaxation. I looked up h-ellipticity of higher order elements and now
> this
> > adds to my worries :(. I may be asking for too much here.... but what do
> > you mean by heavy smoothers? or correction based on a discretization?.
>
> You can use one discretization for evaluating residuals, but then use an
> embedded low-order discretization to apply the correction.  See "defect
> correction" in Achi's multigrid guide or in Trottenberg.
>
> The paper I mentioned earlier was doing something simpler and less
> intrusive: assemble the embedded low-order operator and feed it to
> algebraic multigrid, but evaluate residuals matrix-free using the
> high-order discretization.  But if you have a reasonable geometric
> hierarchy, the defect correction schemes are better.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/d6f1a9bb/attachment.html>

From charlesj at purdue.edu  Fri Oct 18 12:35:53 2013
From: charlesj at purdue.edu (James A Charles)
Date: Fri, 18 Oct 2013 13:35:53 -0400 (EDT)
Subject: [petsc-users] Dense Matrix setting local rows / local columns
 Question
In-Reply-To: <CAMYG4GmpzQj2Opn-kGB=drqViXwg80fNeuUH1rE4SOSxWU0GVg@mail.gmail.com>
Message-ID: <779319987.283947.1382117753650.JavaMail.root@mailhub050.itcs.purdue.edu>

Thank you all for the responses. I was able to figure out the issue with the dense matrix. I had another question related to dense matrices. Does there exist support for multiplication of a sparse A and dense B matrix matrix multiplication in parallel? I'm using Petsc Release Version 3.4.2 and I am getting an error "No support for this operation for this object type! MatMatMult not supported for B of type mpidense" This makes me think that this is not supported. If it's not, are there plans to support this?  

Thanks,
James

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "James A Charles" <charlesj at purdue.edu>
Cc: petsc-users at mcs.anl.gov
Sent: Thursday, October 17, 2013 1:22:36 PM
Subject: Re: [petsc-users] Dense Matrix setting local rows / local columns Question


On Thu, Oct 17, 2013 at 10:02 AM, James A Charles < charlesj at purdue.edu > wrote: 




Hello, 

I'm trying to use a Dense Parallel matrix where there are no nonzeros that is rectangular of size Nxp where N >> p. P is anywhere from 2-6 typically and N can be of size 1E6 or more. 

For this I would like to only distribute the rows along the MPI processes and not have the columns distributed at all (local columns size = global column size). What is the best way to do this in Petsc? If it matters I am using Petsc3.4. 



We do not distribute columns. You could if you used the Elemental implementation, but you do not need that. 


The operations performed after the Matrix allocation are: 

direct call to Lapack for QR factorization via pointer to array. 



You really want Tall-Skinny QR here (TSQR). We have not implemented it, but it is not hard, so if 
you would like to contribute it, that would be great. 


Matrix multiply with an NxN matrix. 



This works. 


Matt 


I have this working in serial but when I try to distribute my matrices I get columns with size p*(number of processes). 

Thanks, 
James 










-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From jedbrown at mcs.anl.gov  Fri Oct 18 12:42:04 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 Oct 2013 12:42:04 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <779319987.283947.1382117753650.JavaMail.root@mailhub050.itcs.purdue.edu>
References: <779319987.283947.1382117753650.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <87haceeas3.fsf@mcs.anl.gov>

James A Charles <charlesj at purdue.edu> writes:

> Thank you all for the responses. I was able to figure out the issue
> with the dense matrix. I had another question related to dense
> matrices. Does there exist support for multiplication of a sparse A
> and dense B matrix matrix multiplication in parallel? I'm using Petsc
> Release Version 3.4.2 and I am getting an error "No support for this
> operation for this object type! MatMatMult not supported for B of type
> mpidense" 

It should be supported.  Please send the entire error message.  (Always
send the entire error message.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/60c148b4/attachment.pgp>

From charlesj at purdue.edu  Fri Oct 18 12:53:19 2013
From: charlesj at purdue.edu (James A Charles)
Date: Fri, 18 Oct 2013 13:53:19 -0400 (EDT)
Subject: [petsc-users] Dense Matrix setting local rows / local columns
 Question
In-Reply-To: <87haceeas3.fsf@mcs.anl.gov>
Message-ID: <662111948.284050.1382118799385.JavaMail.root@mailhub050.itcs.purdue.edu>

My apologies below is the error message:

[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: MatMatMult not supported for B of type mpidense!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./nemo on a linux-static named conte-fe02.rcac.purdue.edu by charlesj Fri Oct 18 10:19:40 2013
[0]PETSC ERROR: Libraries linked from /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/linux-static/lib
[0]PETSC ERROR: Configure run at Wed Aug  7 16:14:36 2013
[0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=complex --with-shared-libraries=0 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-debugging=1 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-mumps=yes --download-hypre=no --download-spooles=yes
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatMatMult() line 8582 in /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/src/mat/interface/matrix.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./nemo on a linux-static named conte-fe02.rcac.purdue.edu by charlesj Fri Oct 18 10:19:40 2013
[0]PETSC ERROR: Libraries linked from /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/linux-static/lib
[0]PETSC ERROR: Configure run at Wed Aug  7 16:14:36 2013
[0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=complex --with-shared-libraries=0 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-debugging=1 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-mumps=yes --download-hypre=no --download-spooles=yes
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetType() line 100 in /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/src/mat/interface/matreg.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./nemo on a linux-static named conte-fe02.rcac.purdue.edu by charlesj Fri Oct 18 10:19:40 2013
[0]PETSC ERROR: Libraries linked from /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/linux-static/lib
[0]PETSC ERROR: Configure run at Wed Aug  7 16:14:36 2013
[0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --download-sowing --with-scalar-type=complex --with-shared-libraries=0 --with-pic=1 --with-clanguage=C++ --with-fortran=1 --with-fortran-kernels=0 --with-debugging=1 --with-blas-lapack-dir=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 --with-blacs-lib=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.so --with-blacs-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --with-scalapack-lib="-L/apps/rhel6/intel/composer_xe_2013.3.163/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/apps/rhel6/intel/composer_xe_2013.3.163/mkl/include --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --download-hdf5=yes --download-metis=yes --download-parmetis=yes --download-superlu_dist=yes --download-mumps=yes --download-hypre=no --download-spooles=yes
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetOwnershipRange() line 6062 in /apps/rhel6/petsc-3.4.2/impi-4.1.0.030_intel-13.1.1.163/src/mat/interface/matrix.c
terminate called after throwing an instance of 'n5_runtime_error'
  what():  [PetscMatrixNemo<cplx>] PETSc gave error with code 62:
    Invalid argument
    .
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)


Thanks,
James
----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "James A Charles" <charlesj at purdue.edu>, "Matthew Knepley" <knepley at gmail.com>
Cc: petsc-users at mcs.anl.gov
Sent: Friday, October 18, 2013 1:42:04 PM
Subject: Re: [petsc-users] Dense Matrix setting local rows / local columns Question

James A Charles <charlesj at purdue.edu> writes:

> Thank you all for the responses. I was able to figure out the issue
> with the dense matrix. I had another question related to dense
> matrices. Does there exist support for multiplication of a sparse A
> and dense B matrix matrix multiplication in parallel? I'm using Petsc
> Release Version 3.4.2 and I am getting an error "No support for this
> operation for this object type! MatMatMult not supported for B of type
> mpidense" 

It should be supported.  Please send the entire error message.  (Always
send the entire error message.)

From charlesj at purdue.edu  Fri Oct 18 13:13:51 2013
From: charlesj at purdue.edu (James A Charles)
Date: Fri, 18 Oct 2013 14:13:51 -0400 (EDT)
Subject: [petsc-users] Dense Matrix setting local rows / local columns
 Question
In-Reply-To: <87haceeas3.fsf@mcs.anl.gov>
Message-ID: <1350731631.284163.1382120031468.JavaMail.root@mailhub050.itcs.purdue.edu>

Actually I think I was wrong where it was occurring (I have several MatMatMults). The MatMatMult for sparse-dense seems to be working! The problem seems to be with dense-dense? I made sure of this by printing out the Matrix Type before calling the multiplication. Is dense-dense not supported?

Thanks,
James 

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "James A Charles" <charlesj at purdue.edu>, "Matthew Knepley" <knepley at gmail.com>
Cc: petsc-users at mcs.anl.gov
Sent: Friday, October 18, 2013 1:42:04 PM
Subject: Re: [petsc-users] Dense Matrix setting local rows / local columns Question

James A Charles <charlesj at purdue.edu> writes:

> Thank you all for the responses. I was able to figure out the issue
> with the dense matrix. I had another question related to dense
> matrices. Does there exist support for multiplication of a sparse A
> and dense B matrix matrix multiplication in parallel? I'm using Petsc
> Release Version 3.4.2 and I am getting an error "No support for this
> operation for this object type! MatMatMult not supported for B of type
> mpidense" 

It should be supported.  Please send the entire error message.  (Always
send the entire error message.)

From jedbrown at mcs.anl.gov  Fri Oct 18 13:27:40 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 Oct 2013 13:27:40 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <1350731631.284163.1382120031468.JavaMail.root@mailhub050.itcs.purdue.edu>
References: <1350731631.284163.1382120031468.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <87eh7ie8o3.fsf@mcs.anl.gov>

James A Charles <charlesj at purdue.edu> writes:

> Actually I think I was wrong where it was occurring (I have several
> MatMatMults). 

If you use the error checking macros (CHKERRQ), you'll get a full stack
trace on errors like that.

> The MatMatMult for sparse-dense seems to be working! The problem seems
> to be with dense-dense? I made sure of this by printing out the Matrix
> Type before calling the multiplication. Is dense-dense not supported?

Correct.  I thought you said your dense matrix had only a few columns.
What are you multiplying it with?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/d560bacf/attachment.pgp>

From jedbrown at mcs.anl.gov  Fri Oct 18 13:29:34 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 Oct 2013 13:29:34 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <87eh7ie8o3.fsf@mcs.anl.gov>
References: <1350731631.284163.1382120031468.JavaMail.root@mailhub050.itcs.purdue.edu>
	<87eh7ie8o3.fsf@mcs.anl.gov>
Message-ID: <87bo2me8kx.fsf@mcs.anl.gov>

Jed Brown <jedbrown at mcs.anl.gov> writes:
>> The MatMatMult for sparse-dense seems to be working! The problem seems
>> to be with dense-dense? I made sure of this by printing out the Matrix
>> Type before calling the multiplication. Is dense-dense not supported?
>
> Correct.  I thought you said your dense matrix had only a few columns.
> What are you multiplying it with?

We used to use PLAPACK for dense-dense products, but it had bugs and we
have since switched to Elemental, which is its own format
(MATELEMENTAL).  But for matrices with only a few columns, you wouldn't
want that distribution.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/45f3f765/attachment.pgp>

From charlesj at purdue.edu  Fri Oct 18 13:35:02 2013
From: charlesj at purdue.edu (James A Charles)
Date: Fri, 18 Oct 2013 14:35:02 -0400 (EDT)
Subject: [petsc-users] Dense Matrix setting local rows / local columns
 Question
In-Reply-To: <87haceeas3.fsf@mcs.anl.gov>
Message-ID: <627472997.284287.1382121302346.JavaMail.root@mailhub050.itcs.purdue.edu>

For this multiplication it is a dense matrix of size Nxp (Where N >> p) with a matrix of size pxp. If dense-sparse works my workaround could be to store the pxp matrix as sparse since it never gets very large anyways (and its triangular). And I don't need direct access to its data array. 

So to summarize what I think I have learned: 
For MatMatMult in parallel sparse-dense, dense-sparse, and sparse-sparse works? And dense-dense does not because you have switched to Elemental and that has a different API that must be used? 

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "James A Charles" <charlesj at purdue.edu>, "Matthew Knepley" <knepley at gmail.com>
Cc: petsc-users at mcs.anl.gov
Sent: Friday, October 18, 2013 1:42:04 PM
Subject: Re: [petsc-users] Dense Matrix setting local rows / local columns Question

James A Charles <charlesj at purdue.edu> writes:

> Thank you all for the responses. I was able to figure out the issue
> with the dense matrix. I had another question related to dense
> matrices. Does there exist support for multiplication of a sparse A
> and dense B matrix matrix multiplication in parallel? I'm using Petsc
> Release Version 3.4.2 and I am getting an error "No support for this
> operation for this object type! MatMatMult not supported for B of type
> mpidense" 

It should be supported.  Please send the entire error message.  (Always
send the entire error message.)

From jedbrown at mcs.anl.gov  Fri Oct 18 13:45:52 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 Oct 2013 13:45:52 -0500
Subject: [petsc-users] Dense Matrix setting local rows / local columns
	Question
In-Reply-To: <627472997.284287.1382121302346.JavaMail.root@mailhub050.itcs.purdue.edu>
References: <627472997.284287.1382121302346.JavaMail.root@mailhub050.itcs.purdue.edu>
Message-ID: <878uxqe7tr.fsf@mcs.anl.gov>

James A Charles <charlesj at purdue.edu> writes:

> For this multiplication it is a dense matrix of size Nxp (Where N >>
> p) with a matrix of size pxp. If dense-sparse works my workaround
> could be to store the pxp matrix as sparse since it never gets very
> large anyways (and its triangular). And I don't need direct access to
> its data array.

Yeah, or you could store the pxp matrix redundantly on each process and
use MatDenseGetArray, then either (a) call BLAS gemm or (b)
MatCreateSeqDense(... with array) and MatMult for the serial matrix
multiply.

> For MatMatMult in parallel sparse-dense, dense-sparse, and
> sparse-sparse works? 

Yes

> And dense-dense does not because you have switched to Elemental and
> that has a different API that must be used?

Well, our elemental support is intended to be simplified and currently
only supports the [MC,MR] distribution, which is a 2D distribution for
squarish matrices.  It would be nice to add support for more Elemental
distributions, especially [VC,*], but I don't have short-term plans to
do this.  Contributions welcome. :-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/7b896f40/attachment.pgp>

From jroman at dsic.upv.es  Fri Oct 18 13:49:00 2013
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 18 Oct 2013 20:49:00 +0200
Subject: [petsc-users] CPU-scaling
In-Reply-To: <CAD6PAKMWR9SqHEicN_k=KWgnoQmftUNgT+SRZa6TYig8zx3YhQ@mail.gmail.com>
References: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
	<0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>
	<CAD6PAKMWR9SqHEicN_k=KWgnoQmftUNgT+SRZa6TYig8zx3YhQ@mail.gmail.com>
Message-ID: <28B6BBAC-5E0D-41AB-B63F-A78E5CE35B00@dsic.upv.es>

No, you don't need MUMPS for ex1.c with default options.
Then I don't know what is happening. Did you build PETSc --with-debugging=0 ? You should send performance data obtained with -log_summary, otherwise we cannot guess what happens.

Jose


El 18/10/2013, a las 17:21, Torquil Macdonald S?rensen escribi?:

> It is a standard eigenvalue problem, and I'm only using default algorithm options, so I don't think I'm using any nontrivial spectral transformation.
> 
> The program is just ex1.c from the documentation, available here:
> 
> http://www.grycap.upv.es/slepc/documentation/current/src/eps/examples/tutorials/ex1.c
> 
> FAQ #6 mentions MUMPS, which I don't have the moment, so I'll have to look into that.
> 
> - Torquil
> 
> 
> 
> On 18 October 2013 16:46, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> El 18/10/2013, a las 16:42, Torquil Macdonald S?rensen escribi?:
> 
> > Hi!
> >
> > I'm just starting to use SLEPc on a quad-core workstation computer (Intel Core i5-2500 CPU @ 3.30GHz).
> >
> > But I'm not getting the speedup that I was expecting, when increasing the number of cores. E.g. when running ex1.c from the documentation, on 1, 2, 3 and 4 cores. When asking for the 500 eigenvalues of smallest magnitude, for a 5000x5000 matrix, I'm getting the following approximate timings:
> >
> > # n_core  time_usage
> > 1              480s
> > 2              400s
> > 3              500s
> > 4              580s
> >
> > Should I not expect a speedup for n_core > 2?
> >
> > The example is using the krylovschur algorithm. For all four runs, the program states "Number of iterations of the method: 33".
> >
> > I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is compiled using GCC 4.8.1 or 4.8.2.
> >
> > Best regards
> > Torquil S?rensen
> >
> 
> Is it a generalized eigenproblem and/or are you doing shift-and-invert? In that case, have a look at FAQ #6 http://www.grycap.upv.es/slepc/documentation/faq.htm
> 
> Jose
> 
> 


From torquil at gmail.com  Fri Oct 18 16:49:08 2013
From: torquil at gmail.com (=?ISO-8859-1?Q?Torquil_Macdonald_S=F8rensen?=)
Date: Fri, 18 Oct 2013 23:49:08 +0200
Subject: [petsc-users] CPU-scaling
In-Reply-To: <28B6BBAC-5E0D-41AB-B63F-A78E5CE35B00@dsic.upv.es>
References: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
	<0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>
	<CAD6PAKMWR9SqHEicN_k=KWgnoQmftUNgT+SRZa6TYig8zx3YhQ@mail.gmail.com>
	<28B6BBAC-5E0D-41AB-B63F-A78E5CE35B00@dsic.upv.es>
Message-ID: <5261ACD4.1000409@gmail.com>

Hi!

You probably hit the nail on the head with the debugging suggestion. 
After reinstalling petsc using --with-debugging=0, I'm getting more 
encouraging results. For the same problem as before, but with a larger 
matrix (now 10000x10000), I'm now getting the following scaling:

# n_core    time (sec)
1                1540
2                870
3                660
4                571

which is much more promising than before. So thanks!

- Torquil

On 18/10/13 20:49, Jose E. Roman wrote:
> No, you don't need MUMPS for ex1.c with default options.
> Then I don't know what is happening. Did you build PETSc --with-debugging=0 ? You should send performance data obtained with -log_summary, otherwise we cannot guess what happens.
>
> Jose
>
>
> El 18/10/2013, a las 17:21, Torquil Macdonald S?rensen escribi?:
>
>> It is a standard eigenvalue problem, and I'm only using default algorithm options, so I don't think I'm using any nontrivial spectral transformation.
>>
>> The program is just ex1.c from the documentation, available here:
>>
>> http://www.grycap.upv.es/slepc/documentation/current/src/eps/examples/tutorials/ex1.c
>>
>> FAQ #6 mentions MUMPS, which I don't have the moment, so I'll have to look into that.
>>
>> - Torquil
>>
>>
>>
>> On 18 October 2013 16:46, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>
>> El 18/10/2013, a las 16:42, Torquil Macdonald S?rensen escribi?:
>>
>>> Hi!
>>>
>>> I'm just starting to use SLEPc on a quad-core workstation computer (Intel Core i5-2500 CPU @ 3.30GHz).
>>>
>>> But I'm not getting the speedup that I was expecting, when increasing the number of cores. E.g. when running ex1.c from the documentation, on 1, 2, 3 and 4 cores. When asking for the 500 eigenvalues of smallest magnitude, for a 5000x5000 matrix, I'm getting the following approximate timings:
>>>
>>> # n_core  time_usage
>>> 1              480s
>>> 2              400s
>>> 3              500s
>>> 4              580s
>>>
>>> Should I not expect a speedup for n_core > 2?
>>>
>>> The example is using the krylovschur algorithm. For all four runs, the program states "Number of iterations of the method: 33".
>>>
>>> I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is compiled using GCC 4.8.1 or 4.8.2.
>>>
>>> Best regards
>>> Torquil S?rensen
>>>
>> Is it a generalized eigenproblem and/or are you doing shift-and-invert? In that case, have a look at FAQ #6 http://www.grycap.upv.es/slepc/documentation/faq.htm
>>
>> Jose
>>
>>


From knepley at gmail.com  Fri Oct 18 16:51:38 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 Oct 2013 16:51:38 -0500
Subject: [petsc-users] CPU-scaling
In-Reply-To: <5261ACD4.1000409@gmail.com>
References: <CAD6PAKNkgOUGm26BcRh5WQP3e0t2U30+etZw6aHRHV1udoDxng@mail.gmail.com>
	<0B262D7D-D8F8-4278-9834-2D41A021FB88@dsic.upv.es>
	<CAD6PAKMWR9SqHEicN_k=KWgnoQmftUNgT+SRZa6TYig8zx3YhQ@mail.gmail.com>
	<28B6BBAC-5E0D-41AB-B63F-A78E5CE35B00@dsic.upv.es>
	<5261ACD4.1000409@gmail.com>
Message-ID: <CAMYG4G=XyXYCqjHbmheOw8CLMK94xo8GZ71Sfp-HXX_TAm-Wuw@mail.gmail.com>

On Fri, Oct 18, 2013 at 4:49 PM, Torquil Macdonald S?rensen <
torquil at gmail.com> wrote:

> Hi!
>
> You probably hit the nail on the head with the debugging suggestion. After
> reinstalling petsc using --with-debugging=0, I'm getting more encouraging
> results. For the same problem as before, but with a larger matrix (now
> 10000x10000), I'm now getting the following scaling:
>
> # n_core    time (sec)
> 1                1540
> 2                870
> 3                660
>

Here you are very likely running into the memory bandwidth bottleneck.
Unfortunately, there is not
a whole lot you can do about this. This is not something vendor's like to
acknowledge.

     Matt


> 4                571
>
> which is much more promising than before. So thanks!
>
> - Torquil
>
> On 18/10/13 20:49, Jose E. Roman wrote:
>
>> No, you don't need MUMPS for ex1.c with default options.
>> Then I don't know what is happening. Did you build PETSc
>> --with-debugging=0 ? You should send performance data obtained with
>> -log_summary, otherwise we cannot guess what happens.
>>
>> Jose
>>
>>
>> El 18/10/2013, a las 17:21, Torquil Macdonald S?rensen escribi?:
>>
>>  It is a standard eigenvalue problem, and I'm only using default
>>> algorithm options, so I don't think I'm using any nontrivial spectral
>>> transformation.
>>>
>>> The program is just ex1.c from the documentation, available here:
>>>
>>> http://www.grycap.upv.es/**slepc/documentation/current/**
>>> src/eps/examples/tutorials/**ex1.c<http://www.grycap.upv.es/slepc/documentation/current/src/eps/examples/tutorials/ex1.c>
>>>
>>> FAQ #6 mentions MUMPS, which I don't have the moment, so I'll have to
>>> look into that.
>>>
>>> - Torquil
>>>
>>>
>>>
>>> On 18 October 2013 16:46, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>>
>>> El 18/10/2013, a las 16:42, Torquil Macdonald S?rensen escribi?:
>>>
>>>  Hi!
>>>>
>>>> I'm just starting to use SLEPc on a quad-core workstation computer
>>>> (Intel Core i5-2500 CPU @ 3.30GHz).
>>>>
>>>> But I'm not getting the speedup that I was expecting, when increasing
>>>> the number of cores. E.g. when running ex1.c from the documentation, on 1,
>>>> 2, 3 and 4 cores. When asking for the 500 eigenvalues of smallest
>>>> magnitude, for a 5000x5000 matrix, I'm getting the following approximate
>>>> timings:
>>>>
>>>> # n_core  time_usage
>>>> 1              480s
>>>> 2              400s
>>>> 3              500s
>>>> 4              580s
>>>>
>>>> Should I not expect a speedup for n_core > 2?
>>>>
>>>> The example is using the krylovschur algorithm. For all four runs, the
>>>> program states "Number of iterations of the method: 33".
>>>>
>>>> I'm using PETSc 3.4.3, SLEPc 3.4.3, OpenMPI 1.6.5, and everything is
>>>> compiled using GCC 4.8.1 or 4.8.2.
>>>>
>>>> Best regards
>>>> Torquil S?rensen
>>>>
>>>>  Is it a generalized eigenproblem and/or are you doing
>>> shift-and-invert? In that case, have a look at FAQ #6
>>> http://www.grycap.upv.es/**slepc/documentation/faq.htm<http://www.grycap.upv.es/slepc/documentation/faq.htm>
>>>
>>> Jose
>>>
>>>
>>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131018/9994a4aa/attachment.html>

From einar.sorheim at gmail.com  Sat Oct 19 14:39:36 2013
From: einar.sorheim at gmail.com (=?ISO-8859-1?Q?Einar_S=F8rheim?=)
Date: Sat, 19 Oct 2013 21:39:36 +0200
Subject: [petsc-users] Problem with GAMG elasticity
 case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
	<6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
Message-ID: <CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>

yes, I call the given code many times, what i have seen so far is that
changing e.g. the order of destruction statements changes the time of
failure,I will try to debug to get more details


2013/10/18 Mark F. Adams <mfadams at lbl.gov>

> I would guess there is some memory corruption or leak someplace.  Try
> running with -on_error_attach_debugger, and try to get some more
> information.  Do you just repeat a call to the code below many times?  Is
> this deterministic, does it fail on the same solve each time?
>
>
> On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com>
> wrote:
>
> Uncommenting  KSPSetFromOptions solved my intial problem, however I got a
> new problem that occurs after many timesteps, in our code we use petsc gamg
> as a solver for several problems(e.g. thermal and mechanical) at each time
> step, I have a suspicion it might have something to do with cleanup after
> solving, the rror appears after many calls to the petsc gamg solver
> interface, the error message is the following:
>
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580
> src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
> Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by
> admeinar Fri Oct 18 12:26:04 2013
> [0]PETSC ERROR: Libraries linked from
> /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes
> --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
> --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
> (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
> --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
> (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>
> The source code for the petsc interface:
>
>       subroutine
> petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond,
> i_Solver
>      $     ,coords_,i_nodedof)
> #include <finclude/petscsys.h>
> #include <finclude/petscvec.h>
> #include <finclude/petscmat.h>
> #include <finclude/petscpc.h>
> #include <finclude/petscksp.h>
> #include <finclude/petscviewer.h>
>
>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
>       integer i_rowstart_(*),i_columnindex_(*)
>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
>       integer, allocatable  :: i_indices_(:)
> !     Variables:
> !
> !  A       - matrix that defines linear system
> !  ksp    - KSP context
> !  ksp     - KSP context
> !  x, b, u - approx solution, RHS, exact solution vectors
> !
> !      implicit none
>       Vec     x,u,b,vec_coords
>       PC      pc
>       Mat     A,F,Matnull
>       KSP    ksp
>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
>       PetscInt Istart,Iend
>       PetscInt nsteps,one, neg_one
>       PetscErrorCode ierr
>       PetscBool  flg
>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
>       PetscOffset i_x
>       PetscViewer viewer
>
>
>       call PetscTime(t0,ierr)
>       write(*,*) "Enter petsc"
>
>       n      = 1
>       nsteps = 1
>       one    = 1
>       neg_one = 1
>       i1 = 1
>       imaxiter = 1200
>
>
>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
>       call MatSetType(A, MATSEQAIJ,ierr)
>       call MatSetUp(A,ierr)
>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
> !  The matrix is partitioned by contiguous chunks of rows across the
> !  processors.  Determine which rows of the matrix are locally owned.
>
>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
>
>
>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
>       do i=1,i_nodes
> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
> !         i_indices_(i+1) = i
> !         write(*,*) "Start of loop2"
>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
> !            write(*,*) "Start of loop 3"
>             v = r_elements_(j)
> !            write(*,*) i1,i,i1,j-1,v
>             call
> MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
>             if (ierr.gt.0) stop
>          end do
>       end do
> !      write(*,*) "End of matrix fill"
>
> !  Assemble matrix, using the 2-step process:
> !       MatAssemblyBegin(), MatAssemblyEnd()
> !  Computations can be done while messages are in transition
> !  by placing code between these two statements.
>
>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>
> !  Create parallel vectors.
> !   - When using VecCreate(), the parallel partitioning of the vector
> !     is determined by PETSc at runtime.
> !   - Note: We form 1 vector from scratch and then duplicate as needed.
> !      write(*,*) "Start of vector fill"
>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
>       call VecSetType(u, VECSEQ,ierr)
>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
> !      call VecSetFromOptions(u,ierr)
>       call VecDuplicate(u,b,ierr)
>       call VecDuplicate(b,x,ierr)
>       do i=1,i_nodes
>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
>       enddo
> !  Assemble vector
>
>        call VecAssemblyBegin(x,ierr)
>        call VecAssemblyEnd(x,ierr)
>        call VecAssemblyBegin(b,ierr)
>        call VecAssemblyEnd(b,ierr)
>        call PetscTime(t1,ierr)
> !       write(*,*) "End of vec fill time spent :", t1-t0
> !  Create linear solver context
>
>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>
>
> !  Set operators. Here the matrix that defines the linear system
> !  also serves as the preconditioning matrix.
>
>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
>
>
>       call KSPGetPC(ksp,pc,ierr)
>       tol = 1.e-20
>       call
> KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>       select case (i_Precond)
>       case ( 1 )
>          call PCSetType(pc,PCJACOBI,ierr)
>       case ( 2 )
>          call PCSetType(pc,PCILU,ierr)
>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
>          call PCFactorSetUpMatSolverPackage(pc,ierr);
>          call PCFactorGetMatrix(pc,F);
>          call MatSuperluSetILUDropTol(F,1.e-4);
>       case ( 3 )
>          call PCSetType(pc,PCGAMG,ierr)
> !         call PCGAMGInitializePackage()
> !         call PCCreate_GAMG(pc,ierr)
>       case DEFAULT
>          call PCSetType(pc,PCJACOBI,ierr)
>       end select
>
> !      call PCSetType(pc,PCJACOBI,ierr)
>       select case (i_solver)
>       case ( 0 )
>          call KSPSetType(ksp,KSPCG,ierr)
>       case DEFAULT
>          call KSPSetType(ksp,KSPCG,ierr)
>       end select
>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>       if (i_nodedof==3) then
>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>          call
> VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>          call MatSetNearNullSpace(A,Matnull,ierr)
>          call MatNullSpaceDestroy(Matnull,ierr)
>
>          write(*,*) "Finish setting null space ierr =", ierr
> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>
>       end if
> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL,
> PETSC_NULL,ierr)
> !  Set runtime options, e.g.,
> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
> !  These options will override those specified above as long as
> !  KSPSetFromOptions() is called _after_ any other customization
> !  routines.
>
>       call KSPSetFromOptions(ksp,ierr)
>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>       call KSPSetUp(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "Time setup", t0-t1
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> !                      Solve the linear system
> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
>       call KSPSolve(ksp,b,x,ierr)
>       call PetscTime(t1,ierr)
>       write(*,*) "end solve, time used:",t1-t0
>       call KSPGetSolution(ksp,x,ierr)
>       call VecGetArray(x,x_array_,i_x,ierr)
>       do i=1,i_nodes
>          r_x_(i) = x_array_(i_x + i)
>       enddo
>       call VecRestoreArray(x,x_array_,i_x,ierr)
>
>
> !  Free work space.  All PETSc objects should be destroyed when they
> !  are no longer needed.
> !      call PCDestroy(pc,ierr)
>
>       call VecDestroy(u,ierr)
>       call VecDestroy(x,ierr)
>       call VecDestroy(b,ierr)
>       call MatDestroy(A,ierr)
>       if (i_nodedof==3) then
>          call VecDestroy(vec_coords,ierr)
>          call MatDestroy(Matnull,ierr)
>       endif
>       call KSPDestroy(ksp,ierr)
>       call PetscTime(t0,ierr)
>       write(*,*) "time celan up :", t0-t1
>
>       end
>
> --
> Einar S?rheim
> <Mail Attachment>
>
>
>


-- 
Einar S?rheim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131019/48787520/attachment.html>

From mfadams at lbl.gov  Sat Oct 19 15:36:06 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Sat, 19 Oct 2013 16:36:06 -0400
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
	<6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
	<CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>
Message-ID: <AE80D149-ED23-4A27-835E-B14208A4F5FC@lbl.gov>

You can try running a problem that finishes with

-malloc
-malloc_debug
-malloc_dump

to see if there are any memory issues.  It looks like you are completely destroying the solver object each time and starting over.  Valgrind would be very useful here because it looks like memory is getting corrupted if the same code runs many times.


On Oct 19, 2013, at 3:39 PM, Einar S?rheim <einar.sorheim at gmail.com> wrote:

> yes, I call the given code many times, what i have seen so far is that changing e.g. the order of destruction statements changes the time of failure,I will try to debug to get more details
> 
> 
> 2013/10/18 Mark F. Adams <mfadams at lbl.gov>
> I would guess there is some memory corruption or leak someplace.  Try running with -on_error_attach_debugger, and try to get some more information.  Do you just repeat a call to the code below many times?  Is this deterministic, does it fail on the same solve each time? 
> 
> 
> On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> 
>> Uncommenting  KSPSetFromOptions solved my intial problem, however I got a new problem that occurs after many timesteps, in our code we use petsc gamg as a solver for several problems(e.g. thermal and mechanical) at each time step, I have a suspicion it might have something to do with cleanup after solving, the rror appears after many calls to the petsc gamg solver interface, the error message is the following:
>> 
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580 src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR: ------------------------------------------------------------------------
>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Fri Oct 18 12:26:04 2013
>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
>> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>> 
>> The source code for the petsc interface: 
>> 
>>       subroutine petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond, i_Solver
>>      $     ,coords_,i_nodedof)
>> #include <finclude/petscsys.h>
>> #include <finclude/petscvec.h>
>> #include <finclude/petscmat.h>
>> #include <finclude/petscpc.h>
>> #include <finclude/petscksp.h>
>> #include <finclude/petscviewer.h>
>> 
>>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
>>       integer i_rowstart_(*),i_columnindex_(*)
>>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
>>       integer, allocatable  :: i_indices_(:)
>> !     Variables:
>> !
>> !  A       - matrix that defines linear system
>> !  ksp    - KSP context
>> !  ksp     - KSP context
>> !  x, b, u - approx solution, RHS, exact solution vectors
>> !
>> !      implicit none
>>       Vec     x,u,b,vec_coords
>>       PC      pc
>>       Mat     A,F,Matnull
>>       KSP    ksp
>>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
>>       PetscInt Istart,Iend
>>       PetscInt nsteps,one, neg_one
>>       PetscErrorCode ierr
>>       PetscBool  flg
>>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
>>       PetscOffset i_x
>>       PetscViewer viewer
>> 
>> 
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Enter petsc"
>> 
>>       n      = 1
>>       nsteps = 1
>>       one    = 1
>>       neg_one = 1
>>       i1 = 1
>>       imaxiter = 1200
>> 
>> 
>>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
>>       call MatSetType(A, MATSEQAIJ,ierr)
>>       call MatSetUp(A,ierr)
>>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
>> !  The matrix is partitioned by contiguous chunks of rows across the
>> !  processors.  Determine which rows of the matrix are locally owned. 
>> 
>>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
>> 
>> 
>>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
>>       do i=1,i_nodes
>> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
>> !         i_indices_(i+1) = i
>> !         write(*,*) "Start of loop2"
>>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
>> !            write(*,*) "Start of loop 3"
>>             v = r_elements_(j)
>> !            write(*,*) i1,i,i1,j-1,v
>>             call MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
>>             if (ierr.gt.0) stop
>>          end do
>>       end do
>> !      write(*,*) "End of matrix fill"
>> 
>> !  Assemble matrix, using the 2-step process:
>> !       MatAssemblyBegin(), MatAssemblyEnd()
>> !  Computations can be done while messages are in transition
>> !  by placing code between these two statements.
>> 
>>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>> 
>> !  Create parallel vectors.
>> !   - When using VecCreate(), the parallel partitioning of the vector
>> !     is determined by PETSc at runtime.
>> !   - Note: We form 1 vector from scratch and then duplicate as needed.
>> !      write(*,*) "Start of vector fill"
>>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
>>       call VecSetType(u, VECSEQ,ierr)
>>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
>> !      call VecSetFromOptions(u,ierr)      
>>       call VecDuplicate(u,b,ierr)
>>       call VecDuplicate(b,x,ierr)
>>       do i=1,i_nodes
>>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
>>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
>>       enddo
>> !  Assemble vector
>> 
>>        call VecAssemblyBegin(x,ierr)
>>        call VecAssemblyEnd(x,ierr)
>>        call VecAssemblyBegin(b,ierr)
>>        call VecAssemblyEnd(b,ierr)
>>        call PetscTime(t1,ierr)
>> !       write(*,*) "End of vec fill time spent :", t1-t0
>> !  Create linear solver context
>> 
>>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>> 
>> 
>> !  Set operators. Here the matrix that defines the linear system
>> !  also serves as the preconditioning matrix.
>> 
>>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
>> 
>> 
>>       call KSPGetPC(ksp,pc,ierr)
>>       tol = 1.e-20
>>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>>       select case (i_Precond)
>>       case ( 1 )
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       case ( 2 )
>>          call PCSetType(pc,PCILU,ierr)
>>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
>>          call PCFactorSetUpMatSolverPackage(pc,ierr); 
>>          call PCFactorGetMatrix(pc,F);
>>          call MatSuperluSetILUDropTol(F,1.e-4);
>>       case ( 3 )
>>          call PCSetType(pc,PCGAMG,ierr)
>> !         call PCGAMGInitializePackage()
>> !         call PCCreate_GAMG(pc,ierr)
>>       case DEFAULT
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       end select
>> 
>> !      call PCSetType(pc,PCJACOBI,ierr)
>>       select case (i_solver)
>>       case ( 0 )
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       case DEFAULT
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       end select
>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>>       if (i_nodedof==3) then
>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>>          call MatSetNearNullSpace(A,Matnull,ierr)
>>          call MatNullSpaceDestroy(Matnull,ierr)
>> 
>>          write(*,*) "Finish setting null space ierr =", ierr
>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>> 
>>       end if
>> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL, PETSC_NULL,ierr)
>> !  Set runtime options, e.g.,
>> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
>> !  These options will override those specified above as long as
>> !  KSPSetFromOptions() is called _after_ any other customization
>> !  routines.
>> 
>>       call KSPSetFromOptions(ksp,ierr)
>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>       call KSPSetUp(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Time setup", t0-t1
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>> !                      Solve the linear system
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>> 
>>       call KSPSolve(ksp,b,x,ierr)
>>       call PetscTime(t1,ierr)
>>       write(*,*) "end solve, time used:",t1-t0
>>       call KSPGetSolution(ksp,x,ierr)
>>       call VecGetArray(x,x_array_,i_x,ierr)
>>       do i=1,i_nodes
>>          r_x_(i) = x_array_(i_x + i)
>>       enddo
>>       call VecRestoreArray(x,x_array_,i_x,ierr)
>> 
>> 
>> !  Free work space.  All PETSc objects should be destroyed when they
>> !  are no longer needed.
>> !      call PCDestroy(pc,ierr)
>> 
>>       call VecDestroy(u,ierr)
>>       call VecDestroy(x,ierr)
>>       call VecDestroy(b,ierr)
>>       call MatDestroy(A,ierr)
>>       if (i_nodedof==3) then
>>          call VecDestroy(vec_coords,ierr)
>>          call MatDestroy(Matnull,ierr)
>>       endif
>>       call KSPDestroy(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "time celan up :", t0-t1
>> 
>>       end
>> 
>> -- 
>> Einar S?rheim
>> <Mail Attachment>
> 
> 
> 
> 
> -- 
> Einar S?rheim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131019/334d135b/attachment-0001.html>

From mfadams at lbl.gov  Sat Oct 19 18:09:42 2013
From: mfadams at lbl.gov (Mark F. Adams)
Date: Sat, 19 Oct 2013 19:09:42 -0400
Subject: [petsc-users] Making a FD Jacobian
Message-ID: <55C4BCE0-6A52-441F-BFD9-5B14BC34BE85@lbl.gov>

I'm trying to make a Jacobian and have this code working:

  ierr = DMCreateMatrix(dm, &A);CHKERRQ(ierr);

  { /* finite difference Jacobian A */
    SNES snes;
    MatStructure flag;

    ierr = SNESCreate(comm, &snes);CHKERRQ(ierr);
    ierr = SNESSetDM(snes,dm);CHKERRQ(ierr);
    ierr = SNESSetFunction(snes,NULL,ComputeFunctionLap,user);CHKERRQ(ierr);
    ierr = SNESSetJacobian(snes,A,A,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr);

    ierr = SNESComputeJacobian(snes,X,&A,&A,&flag);CHKERRQ(ierr);
    /* ierr = MatView(A,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); */
    ierr = SNESDestroy(&snes);CHKERRQ(ierr);
  }

except I need to comment out these line in snesj2.c line ~75:

  ierr = VecEqual(x1,snes->vec_sol,&solvec);CHKERRQ(ierr);
  if (!snes->vec_rhs && solvec) {
    ierr = MatFDColoringSetF(color,F);CHKERRQ(ierr);
  }
  
The problem is that snes->vec_sol is NULL and VecEqual asserts that these are valid arguments.  How should I go about doing this?

Mark


From fande.kong at colorado.edu  Sat Oct 19 19:34:08 2013
From: fande.kong at colorado.edu (Fande Kong)
Date: Sat, 19 Oct 2013 18:34:08 -0600
Subject: [petsc-users] How to update current version of the petsc to the
	last version ?
Message-ID: <CAN5Wd-L0gJRDBHVntTGRN7PgZhK_4PBDY-c3Cw4EmRY4NZDr1A@mail.gmail.com>

Hi all,

I am using petsc-3.4.1, and I want to update the current version to the
last version. How to do that? Just delete it, and then reinstall the new
petsc?


Thanks,

Fande Kong,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131019/4cfb187d/attachment.html>

From jedbrown at mcs.anl.gov  Sat Oct 19 19:48:33 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 19 Oct 2013 19:48:33 -0500
Subject: [petsc-users] How to update current version of the petsc to the
	last version ?
In-Reply-To: <CAN5Wd-L0gJRDBHVntTGRN7PgZhK_4PBDY-c3Cw4EmRY4NZDr1A@mail.gmail.com>
References: <CAN5Wd-L0gJRDBHVntTGRN7PgZhK_4PBDY-c3Cw4EmRY4NZDr1A@mail.gmail.com>
Message-ID: <87mwm4bwda.fsf@mcs.anl.gov>

Fande Kong <fande.kong at colorado.edu> writes:

> Hi all,
>
> I am using petsc-3.4.1, and I want to update the current version to the
> last version. How to do that? Just delete it, and then reinstall the new
> petsc?

Yes.  Or, if you obtained v3.4.1 from Git, you could

  $ git checkout maint
  $ git pull
  $ $PETSC_ARCH/conf/reconfigure*.py

http://www.mcs.anl.gov/petsc/download/index.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131019/f8334448/attachment.pgp>

From jedbrown at mcs.anl.gov  Sat Oct 19 22:58:27 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 19 Oct 2013 22:58:27 -0500
Subject: [petsc-users] Making a FD Jacobian
In-Reply-To: <55C4BCE0-6A52-441F-BFD9-5B14BC34BE85@lbl.gov>
References: <55C4BCE0-6A52-441F-BFD9-5B14BC34BE85@lbl.gov>
Message-ID: <87haccbnks.fsf@mcs.anl.gov>

"Mark F. Adams" <mfadams at lbl.gov> writes:

> I'm trying to make a Jacobian and have this code working:
>
>   ierr = DMCreateMatrix(dm, &A);CHKERRQ(ierr);
>
>   { /* finite difference Jacobian A */
>     SNES snes;
>     MatStructure flag;
>
>     ierr = SNESCreate(comm, &snes);CHKERRQ(ierr);
>     ierr = SNESSetDM(snes,dm);CHKERRQ(ierr);
>     ierr = SNESSetFunction(snes,NULL,ComputeFunctionLap,user);CHKERRQ(ierr);
>     ierr = SNESSetJacobian(snes,A,A,SNESComputeJacobianDefaultColor,0);CHKERRQ(ierr);
>
>     ierr = SNESComputeJacobian(snes,X,&A,&A,&flag);CHKERRQ(ierr);

The API normally requires that the residual be evaluated before the
Jacobian, so that users can compute any part of the Jacobian during
residual evaluation.  That assumption could be weakened slightly if
necessary, but if you're writing library code, that would need to be a
conscious decision.

>     /* ierr = MatView(A,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); */
>     ierr = SNESDestroy(&snes);CHKERRQ(ierr);
>   }
>
> except I need to comment out these line in snesj2.c line ~75:
>
>   ierr = VecEqual(x1,snes->vec_sol,&solvec);CHKERRQ(ierr);

You could just skip this fast path if snes->vec_sol is NULL.

>   if (!snes->vec_rhs && solvec) {
>     ierr = MatFDColoringSetF(color,F);CHKERRQ(ierr);
>   }
>   
> The problem is that snes->vec_sol is NULL and VecEqual asserts that these are valid arguments.  How should I go about doing this?
>
> Mark
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131019/a1a6ce04/attachment.pgp>

From fd.kong at siat.ac.cn  Sun Oct 20 15:04:39 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 20 Oct 2013 14:04:39 -0600
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
Message-ID: <CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>

This behaviour is really, really strange.

The yellowstone supercomputer updated the IBM PE to version 1.3.0.4 about
two months ago (
https://dailyb.cisl.ucar.edu/bulletins/yellowstone-outage-august-27-update-ibm-pe-and-lsf).
I recompiled the petsc and my code as they suggested. Unfortunately, this
problem reoccurs even with small the number of processors (512 cores).

The problem only happened before with large the number of processors and
large the size of problem, but now this problem occurs even with small the
number of processors and small problem for using IBM MPI or intel MPI.

The exactly same code can run on another supercomputer. I think the code
matstash.c is really sensitive on the IBM PE. It is hard for me to fix it.
Can we disable stash and then I send off-processors data by ourselves? Or
can we attach a scatter to mat to exchange off-processors values?

error messages:

[76]PETSC ERROR: --------------------- Error Message
------------------------------------
[76]PETSC ERROR: Petsc has generated inconsistent data!
[76]PETSC ERROR: Negative MPI source: stash->nrecvs=5 i=7 MPI_SOURCE=-32766
MPI_TAG=-32766 MPI_ERROR=371173!
[76]PETSC ERROR:
------------------------------------------------------------------------
[76]PETSC ERROR: Petsc Release Version 3.4.1, unknown
[76]PETSC ERROR: See docs/changes/index.html for recent updates.
[76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[76]PETSC ERROR: See docs/index.html for manual pages.
[76]PETSC ERROR:
------------------------------------------------------------------------
[76]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named ys0623
by fandek Sat Oct 19 00:26:16 2013
[76]PETSC ERROR: Libraries linked from
/glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib
[76]PETSC ERROR: Configure run at Fri Oct 18 23:57:35 2013
[76]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx
--with-shared-libraries=1 --with-dynamic-loading=1
--download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1
--download-metis=1 --with-64-bit-indices=1 --download-netcdf=1
--download-exodusii=1 --download-ptscotch=1 --download-hdf5=1
--with-debugging=no
[76]PETSC ERROR:
------------------------------------------------------------------------
[76]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in
/glade/p/work/fandek/petsc/src/mat/utils/matstash.c
[76]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in
/glade/p/work/fandek/petsc/src/mat/impls/aij/mpi/mpiaij.c
[76]PETSC ERROR: MatAssemblyEnd() line 4939 in
/glade/p/work/fandek/petsc/src/mat/interface/matrix.c
[76]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp
[76]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in
meshreorder.cpp
[76]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp
[76]PETSC ERROR: DMmeshInitialize() line 95 in mgInitialize.cpp
[76]PETSC ERROR: main() line 69 in linearElasticity3d.cpp
Abort(77) on node 76 (rank 76 in comm 1140850688): application called
MPI_Abort(MPI_COMM_WORLD, 77) - process 76
ERROR: 0031-300  Forcing all remote tasks to exit due to exit code 1 in
task 76


Thanks



On Wed, Jun 26, 2013 at 8:56 AM, Jeff Hammond <jhammond at alcf.anl.gov> wrote:

> This concerns IBM PE-MPI on iDataPlex, which is likely based upon the
> cluster implementation of PAMI, which is a completely different code base
> from the PAMI Blue Gene implementation.  If you can reproduce it on Blue
> Gene/Q, I will care.
>
> As an IBM customer, NCAR is endowed with the ability to file bug reports
> directly with IBM related to the products they possess.  There is a link to
> their support system on http://www2.cisl.ucar.edu/resources/yellowstone,
> which is the appropriate channel for users of Yellowstone that have issues
> with the system software installed there.
>
> Jeff
>
> ----- Original Message -----
> From: "Jed Brown" <jedbrown at mcs.anl.gov>
> To: "Fande Kong" <fd.kong at siat.ac.cn>, "petsc-users" <
> petsc-users at mcs.anl.gov>
> Cc: "Jeff Hammond" <jhammond at alcf.anl.gov>
> Sent: Wednesday, June 26, 2013 9:21:48 AM
> Subject: Re: [petsc-users] How to understand these error messages
>
> Fande Kong <fd.kong at siat.ac.cn> writes:
>
> > Hi Barry,
> >
> > If I use the intel mpi, my code can correctly run and can produce some
> > correct results. Yes, you are right. The IBM MPI has some bugs.
>
> Fande, please report this issue to the IBM.
>
> Jeff, Fande has a reproducible case where when running on 10k cores and
> problem sizes over 100M, this
>
>   MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);
>
> returns
>
>       [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11
>       MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892!
>
> It runs correctly for smaller problem sizes, smaller core counts, or for
> all sizes when using Intel MPI.  This is on Yellowstone (iDataPlex, 4500
> dx360 nodes).  Do you know someone at IBM that should be notified?
>
> --
> Jeff Hammond
> Argonne Leadership Computing Facility
> University of Chicago Computation Institute
> jhammond at alcf.anl.gov / (630) 252-5381
> http://www.linkedin.com/in/jeffhammond
> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> ALCF docs: http://www.alcf.anl.gov/user-guides
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131020/c22e546a/attachment.html>

From bsmith at mcs.anl.gov  Sun Oct 20 15:15:57 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 20 Oct 2013 15:15:57 -0500
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
Message-ID: <DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>


  It is unfortunate IBM has perpetuated this error in their libraries and made it worse. 

   You can, of course, work around it by making your application code far more complicated and manage the matrix assembly yourself but that is not a good use of your time or anyone's time. Plus how do you know that their bugging MPI won't bite you somewhere else, like in the new code you would need to write.

   You need to report this bug to IBM and they need to take it seriously, unfortunately if you are not the purchaser of the IBM you are running on they may not care (companies only care about paying customers who complain). 

   Can you just use the Intel MPI mpi compilers/libraries? Or switch to some other system that is not from IBM? Better to just not use this machine until IBM straightens it out.

   Barry



On Oct 20, 2013, at 3:04 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:

> This behaviour is really, really strange.
> 
> The yellowstone supercomputer updated the IBM PE to version 1.3.0.4 about two months ago (https://dailyb.cisl.ucar.edu/bulletins/yellowstone-outage-august-27-update-ibm-pe-and-lsf). I recompiled the petsc and my code as they suggested. Unfortunately, this problem reoccurs even with small the number of processors (512 cores).
> 
> The problem only happened before with large the number of processors and large the size of problem, but now this problem occurs even with small the number of processors and small problem for using IBM MPI or intel MPI.
> 
> The exactly same code can run on another supercomputer. I think the code matstash.c is really sensitive on the IBM PE. It is hard for me to fix it. Can we disable stash and then I send off-processors data by ourselves? Or can we attach a scatter to mat to exchange off-processors values?
> 
> error messages:
> 
> [76]PETSC ERROR: --------------------- Error Message ------------------------------------
> [76]PETSC ERROR: Petsc has generated inconsistent data!
> [76]PETSC ERROR: Negative MPI source: stash->nrecvs=5 i=7 MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=371173!
> [76]PETSC ERROR: ------------------------------------------------------------------------
> [76]PETSC ERROR: Petsc Release Version 3.4.1, unknown 
> [76]PETSC ERROR: See docs/changes/index.html for recent updates.
> [76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [76]PETSC ERROR: See docs/index.html for manual pages.
> [76]PETSC ERROR: ------------------------------------------------------------------------
> [76]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named ys0623 by fandek Sat Oct 19 00:26:16 2013
> [76]PETSC ERROR: Libraries linked from /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib
> [76]PETSC ERROR: Configure run at Fri Oct 18 23:57:35 2013
> [76]PETSC ERROR: Configure options --with-valgrind=1 --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1 --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1 --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1 --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1 --with-debugging=no
> [76]PETSC ERROR: ------------------------------------------------------------------------
> [76]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in /glade/p/work/fandek/petsc/src/mat/utils/matstash.c
> [76]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in /glade/p/work/fandek/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [76]PETSC ERROR: MatAssemblyEnd() line 4939 in /glade/p/work/fandek/petsc/src/mat/interface/matrix.c
> [76]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in meshreorder.cpp
> [76]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in meshreorder.cpp
> [76]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp
> [76]PETSC ERROR: DMmeshInitialize() line 95 in mgInitialize.cpp
> [76]PETSC ERROR: main() line 69 in linearElasticity3d.cpp
> Abort(77) on node 76 (rank 76 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 77) - process 76
> ERROR: 0031-300  Forcing all remote tasks to exit due to exit code 1 in task 76
> 
> 
> Thanks 
> 
> 
> 
> On Wed, Jun 26, 2013 at 8:56 AM, Jeff Hammond <jhammond at alcf.anl.gov> wrote:
> This concerns IBM PE-MPI on iDataPlex, which is likely based upon the cluster implementation of PAMI, which is a completely different code base from the PAMI Blue Gene implementation.  If you can reproduce it on Blue Gene/Q, I will care.
> 
> As an IBM customer, NCAR is endowed with the ability to file bug reports directly with IBM related to the products they possess.  There is a link to their support system on http://www2.cisl.ucar.edu/resources/yellowstone, which is the appropriate channel for users of Yellowstone that have issues with the system software installed there.
> 
> Jeff
> 
> ----- Original Message -----
> From: "Jed Brown" <jedbrown at mcs.anl.gov>
> To: "Fande Kong" <fd.kong at siat.ac.cn>, "petsc-users" <petsc-users at mcs.anl.gov>
> Cc: "Jeff Hammond" <jhammond at alcf.anl.gov>
> Sent: Wednesday, June 26, 2013 9:21:48 AM
> Subject: Re: [petsc-users] How to understand these error messages
> 
> Fande Kong <fd.kong at siat.ac.cn> writes:
> 
> > Hi Barry,
> >
> > If I use the intel mpi, my code can correctly run and can produce some
> > correct results. Yes, you are right. The IBM MPI has some bugs.
> 
> Fande, please report this issue to the IBM.
> 
> Jeff, Fande has a reproducible case where when running on 10k cores and
> problem sizes over 100M, this
> 
>   MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);
> 
> returns
> 
>       [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11
>       MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892!
> 
> It runs correctly for smaller problem sizes, smaller core counts, or for
> all sizes when using Intel MPI.  This is on Yellowstone (iDataPlex, 4500
> dx360 nodes).  Do you know someone at IBM that should be notified?
> 
> --
> Jeff Hammond
> Argonne Leadership Computing Facility
> University of Chicago Computation Institute
> jhammond at alcf.anl.gov / (630) 252-5381
> http://www.linkedin.com/in/jeffhammond
> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> ALCF docs: http://www.alcf.anl.gov/user-guides
> 
> 


From fd.kong at siat.ac.cn  Sun Oct 20 15:51:24 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Sun, 20 Oct 2013 14:51:24 -0600
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
	<DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
Message-ID: <CAN5Wd-+WoG5J706GFuMwpadMhmCCtfGU_q9aRSEW7Pw=b78RYg@mail.gmail.com>

I can use intel MPI and compilers but still have the same problem since all
the mpi programs need to run on the IBM parallel environment.
http://www-03.ibm.com/systems/software/parallel/.


On Sun, Oct 20, 2013 at 2:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   It is unfortunate IBM has perpetuated this error in their libraries and
> made it worse.
>
>    You can, of course, work around it by making your application code far
> more complicated and manage the matrix assembly yourself but that is not a
> good use of your time or anyone's time. Plus how do you know that their
> bugging MPI won't bite you somewhere else, like in the new code you would
> need to write.
>
>    You need to report this bug to IBM and they need to take it seriously,
> unfortunately if you are not the purchaser of the IBM you are running on
> they may not care (companies only care about paying customers who complain).
>
>    Can you just use the Intel MPI mpi compilers/libraries? Or switch to
> some other system that is not from IBM? Better to just not use this machine
> until IBM straightens it out.
>
>    Barry
>
>
>
> On Oct 20, 2013, at 3:04 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>
> > This behaviour is really, really strange.
> >
> > The yellowstone supercomputer updated the IBM PE to version 1.3.0.4
> about two months ago (
> https://dailyb.cisl.ucar.edu/bulletins/yellowstone-outage-august-27-update-ibm-pe-and-lsf).
> I recompiled the petsc and my code as they suggested. Unfortunately, this
> problem reoccurs even with small the number of processors (512 cores).
> >
> > The problem only happened before with large the number of processors and
> large the size of problem, but now this problem occurs even with small the
> number of processors and small problem for using IBM MPI or intel MPI.
> >
> > The exactly same code can run on another supercomputer. I think the code
> matstash.c is really sensitive on the IBM PE. It is hard for me to fix it.
> Can we disable stash and then I send off-processors data by ourselves? Or
> can we attach a scatter to mat to exchange off-processors values?
> >
> > error messages:
> >
> > [76]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [76]PETSC ERROR: Petsc has generated inconsistent data!
> > [76]PETSC ERROR: Negative MPI source: stash->nrecvs=5 i=7
> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=371173!
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: Petsc Release Version 3.4.1, unknown
> > [76]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [76]PETSC ERROR: See docs/index.html for manual pages.
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named
> ys0623 by fandek Sat Oct 19 00:26:16 2013
> > [76]PETSC ERROR: Libraries linked from
> /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib
> > [76]PETSC ERROR: Configure run at Fri Oct 18 23:57:35 2013
> > [76]PETSC ERROR: Configure options --with-valgrind=1
> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1
> --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1
> --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1
> --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1
> --with-debugging=no
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in
> /glade/p/work/fandek/petsc/src/mat/utils/matstash.c
> > [76]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in
> /glade/p/work/fandek/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [76]PETSC ERROR: MatAssemblyEnd() line 4939 in
> /glade/p/work/fandek/petsc/src/mat/interface/matrix.c
> > [76]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in
> meshreorder.cpp
> > [76]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in
> meshreorder.cpp
> > [76]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp
> > [76]PETSC ERROR: DMmeshInitialize() line 95 in mgInitialize.cpp
> > [76]PETSC ERROR: main() line 69 in linearElasticity3d.cpp
> > Abort(77) on node 76 (rank 76 in comm 1140850688): application called
> MPI_Abort(MPI_COMM_WORLD, 77) - process 76
> > ERROR: 0031-300  Forcing all remote tasks to exit due to exit code 1 in
> task 76
> >
> >
> > Thanks
> >
> >
> >
> > On Wed, Jun 26, 2013 at 8:56 AM, Jeff Hammond <jhammond at alcf.anl.gov>
> wrote:
> > This concerns IBM PE-MPI on iDataPlex, which is likely based upon the
> cluster implementation of PAMI, which is a completely different code base
> from the PAMI Blue Gene implementation.  If you can reproduce it on Blue
> Gene/Q, I will care.
> >
> > As an IBM customer, NCAR is endowed with the ability to file bug reports
> directly with IBM related to the products they possess.  There is a link to
> their support system on http://www2.cisl.ucar.edu/resources/yellowstone,
> which is the appropriate channel for users of Yellowstone that have issues
> with the system software installed there.
> >
> > Jeff
> >
> > ----- Original Message -----
> > From: "Jed Brown" <jedbrown at mcs.anl.gov>
> > To: "Fande Kong" <fd.kong at siat.ac.cn>, "petsc-users" <
> petsc-users at mcs.anl.gov>
> > Cc: "Jeff Hammond" <jhammond at alcf.anl.gov>
> > Sent: Wednesday, June 26, 2013 9:21:48 AM
> > Subject: Re: [petsc-users] How to understand these error messages
> >
> > Fande Kong <fd.kong at siat.ac.cn> writes:
> >
> > > Hi Barry,
> > >
> > > If I use the intel mpi, my code can correctly run and can produce some
> > > correct results. Yes, you are right. The IBM MPI has some bugs.
> >
> > Fande, please report this issue to the IBM.
> >
> > Jeff, Fande has a reproducible case where when running on 10k cores and
> > problem sizes over 100M, this
> >
> >   MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);
> >
> > returns
> >
> >       [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11
> >       MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892!
> >
> > It runs correctly for smaller problem sizes, smaller core counts, or for
> > all sizes when using Intel MPI.  This is on Yellowstone (iDataPlex, 4500
> > dx360 nodes).  Do you know someone at IBM that should be notified?
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > University of Chicago Computation Institute
> > jhammond at alcf.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> > ALCF docs: http://www.alcf.anl.gov/user-guides
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131020/36db8f2e/attachment-0001.html>

From meng.wu at inria.fr  Mon Oct 21 04:39:35 2013
From: meng.wu at inria.fr (Meng Wu)
Date: Mon, 21 Oct 2013 11:39:35 +0200 (CEST)
Subject: [petsc-users] :)
In-Reply-To: <439786897.505138.1382345606408.JavaMail.root@inria.fr>
Message-ID: <414488736.523911.1382348375120.JavaMail.root@inria.fr>

Dear Sir or Madam, 
I met a problem during using PETSc. 
/usr/bin/ld: /user/mewu/home/PETSc/arch-linux2-c-debug/lib/libpetsc.a(pinit.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: /user/mewu/home/PETSc/arch-linux2-c-debug/lib/libpetsc.a(pinit.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC 
/user/mewu/home/PETSc/arch-linux2-c-debug/lib/libpetsc.a: could not read symbols: Bad value 
collect2: ld returned 1 exit state 

I have recompiled PETSc with? --cppflags="-fPIC". 
However, the problem is still there. 

Could you give me some suggestions? 

Best, 

Meng 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/d29d5677/attachment.html>

From rupp at mcs.anl.gov  Mon Oct 21 05:13:18 2013
From: rupp at mcs.anl.gov (Karl Rupp)
Date: Mon, 21 Oct 2013 12:13:18 +0200
Subject: [petsc-users] [petsc-maint] :)
In-Reply-To: <439786897.505138.1382345606408.JavaMail.root@inria.fr>
References: <439786897.505138.1382345606408.JavaMail.root@inria.fr>
Message-ID: <5264FE3E.8050006@mcs.anl.gov>

Hi Meng,

please send the full error messages (configure.log, error.log). -fPIC is 
supposed to be set by default.

Best regards,
Karli


On 10/21/2013 10:53 AM, Meng Wu wrote:
>
> Dear Sir or Madam,
> I met a problem during using PETSc.
>
> /usr/bin/ld:
> /user/mewu/home/PETSc/arch-linux2-c-debug/lib/libpetsc.a(pinit.c.o):
> relocation R_X86_64_32S against `.rodata' can not be used when making a
> shared object; recompile with -fPIC
>
>
> /user/mewu/home/PETSc/arch-linux2-c-debug/lib/libpetsc.a: could not read
> symbols: Bad value
>
> collect2: ld returned 1 exit state
>
> I have recompiled PETSc with  --cppflags="-fPIC".
> However, the problem is still there.
>
> Could you give me some suggestions?
>
> Best,
>
> Meng


From asmund.ervik at ntnu.no  Mon Oct 21 06:40:06 2013
From: asmund.ervik at ntnu.no (=?ISO-8859-1?Q?=C5smund_Ervik?=)
Date: Mon, 21 Oct 2013 13:40:06 +0200
Subject: [petsc-users] Getting DMDA local coordinates
Message-ID: <52651296.6030102@ntnu.no>

Dear all,

I'm using a DMDA to manage my grid, and I want to have uniform
coordinates, so I use DMDASetUniformCoordinates. Since I let PETSc
decide the grid partitioning, I need to get the local coordinates, i.e.
xmin,xmax for the local grid. I found the function
DMGetCoordinatesLocal, but I can't seem to figure out how it works (not
in the manual, examples are confusing). What I do is

  call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
  call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
  call DMDAVecGetArrayF90(SolScal,coordVec,coords,ierr)

Then I have the "coords" F90 array, which has the same dimensions as my
local part of the DA, i.e. dof:imax:jmax:kmax (all starting at 0). Where
in this array can I find the values xmin,xmax etc? It looks really
strange, e.g. a slice in i-direction gives
 0.10
 0.00
 1.20
 0.55
 0.00

I do realize I can compute these on my own, but I think using
DMGetCoordinatesLocal has less potential for errors.

Best regards,
?smund Ervik

From jedbrown at mcs.anl.gov  Mon Oct 21 06:55:39 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 Oct 2013 06:55:39 -0500
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <52651296.6030102@ntnu.no>
References: <52651296.6030102@ntnu.no>
Message-ID: <87sivuzvlw.fsf@mcs.anl.gov>

?smund Ervik <asmund.ervik at ntnu.no> writes:

> Dear all,
>
> I'm using a DMDA to manage my grid, and I want to have uniform
> coordinates, so I use DMDASetUniformCoordinates. Since I let PETSc
> decide the grid partitioning, I need to get the local coordinates, i.e.
> xmin,xmax for the local grid. I found the function
> DMGetCoordinatesLocal, but I can't seem to figure out how it works (not
> in the manual, examples are confusing). What I do is
>
>   call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
>   call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
>   call DMDAVecGetArrayF90(SolScal,coordVec,coords,ierr)
>
> Then I have the "coords" F90 array, which has the same dimensions as my
> local part of the DA, i.e. dof:imax:jmax:kmax (all starting at 0). 

Actually, the dimensions are dim:imax:jmax:kmax.  Use

  call DMGetCoordinateDM(SolScal,CoordDM,ierr)
  call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)

> Where in this array can I find the values xmin,xmax etc? It looks
> really strange, e.g. a slice in i-direction gives 0.10 0.00 1.20 0.55
> 0.00
>
> I do realize I can compute these on my own, but I think using
> DMGetCoordinatesLocal has less potential for errors.
>
> Best regards,
> ?smund Ervik
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/482ca565/attachment.pgp>

From juhaj at iki.fi  Mon Oct 21 07:02:17 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Mon, 21 Oct 2013 14:02:17 +0200
Subject: [petsc-users] memory use of a DMDA
Message-ID: <3178358.Nb05eOjZJN@dhcp071>

Dear list members,

I have noticed strange memory consumption after upgrading to 3.4 series. I 
never had time to properly investigate, but here is what happens [yes, this 
might be a petsc4py issue, but I doubt it] is

# helpers contains _ProcessMemoryInfoProc routine which just digs the memory
# usage data from /proc
import helpers
procdata=helpers._ProcessMemoryInfoProc()
print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
from petsc4py import PETSc
procdata=helpers._ProcessMemoryInfoProc()
print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
da = PETSc.DA().create(sizes=[100,100,100],
                       proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
                       boundary_type=[3,0,0],
                       stencil_type=PETSc.DA.StencilType.BOX,
                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
procdata=helpers._ProcessMemoryInfoProc()
print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
vec=da.createGlobalVec()
procdata=helpers._ProcessMemoryInfoProc()
print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]

outputs

48 MiB / 49348 kB
48 MiB / 49360 kB
381 MiB / 446228 kB
435 MiB / 446228 kB

Which is odd: size of the actual data to be stored in the da is just about 56 
megabytes, so why does creating the da consume 7 times that? And why does the 
DA reserve the memory in the first place? I thought memory only gets allocated 
once an associated vector is created and it indeed looks like the 
createGlobalVec call does indeed allocate the right amount of data. But what 
is that 330 MiB that DA().create() consumes? [It's actually the .setUp() 
method that does the consuming, but that's not of much use as it needs to be 
called before a vector can be created.]

Cheers,
Juha


From asmund.ervik at ntnu.no  Mon Oct 21 07:14:52 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Mon, 21 Oct 2013 14:14:52 +0200
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <87sivuzvlw.fsf@mcs.anl.gov>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
Message-ID: <52651ABC.1060306@ntnu.no>

Thanks Jed,

The following works fine (for future reference):

  call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)

  call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
  call DMGetCoordinateDM(SolScal,CoordDM,ierr)
  call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)

  write(*,*) coords(:,0,0,0)
  write(*,*) coords(:,imax-1,jmax-1,kmax-1)
gives
  0.1  0.0  0.0
  1.0  1.2  1.0

Perfect. Not very logical though, that we need this new DM, so perhaps
add an example/something in the manual?


I'm by the way planning to send you a Fortran example solving van der
Pol with a 3D DMDA (if you want it). I could include the above code in
that example if desired.  The example is written with some utility
functions (discussed in emails a few weeks ago) such that it is easy to
reuse 95% of a legacy code solving van der Pol (or other PDE). The main
motivation for adding this example (IMO) is that none of the current DM
examples show how DMDAs are used for solving a PDE in a very simple
case, particularly not for Fortran.

One question in this regard: can you have examples split into two files?
That makes it easier to use Fortran modules in this case.


?smund


On 21. okt. 2013 13:55, Jed Brown wrote:
> ?smund Ervik <asmund.ervik at ntnu.no> writes:
> 
>> Dear all,
>>
>> I'm using a DMDA to manage my grid, and I want to have uniform
>> coordinates, so I use DMDASetUniformCoordinates. Since I let PETSc
>> decide the grid partitioning, I need to get the local coordinates, i.e.
>> xmin,xmax for the local grid. I found the function
>> DMGetCoordinatesLocal, but I can't seem to figure out how it works (not
>> in the manual, examples are confusing). What I do is
>>
>>   call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
>>   call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
>>   call DMDAVecGetArrayF90(SolScal,coordVec,coords,ierr)
>>
>> Then I have the "coords" F90 array, which has the same dimensions as my
>> local part of the DA, i.e. dof:imax:jmax:kmax (all starting at 0). 
> 
> Actually, the dimensions are dim:imax:jmax:kmax.  Use
> 
>   call DMGetCoordinateDM(SolScal,CoordDM,ierr)
>   call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)
> 
>> Where in this array can I find the values xmin,xmax etc? It looks
>> really strange, e.g. a slice in i-direction gives 0.10 0.00 1.20 0.55
>> 0.00
>>
>> I do realize I can compute these on my own, but I think using
>> DMGetCoordinatesLocal has less potential for errors.
>>
>> Best regards,
>> ?smund Ervik

From knepley at gmail.com  Mon Oct 21 07:41:20 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 07:41:20 -0500
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <52651ABC.1060306@ntnu.no>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no>
Message-ID: <CAMYG4GmwviAUNHW5XZmsHwgdON9TnaDK-wSxVuRm=n9fM8Y+RQ@mail.gmail.com>

On Mon, Oct 21, 2013 at 7:14 AM, ?smund Ervik <asmund.ervik at ntnu.no> wrote:

> Thanks Jed,
>
> The following works fine (for future reference):
>
>   call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
>
>   call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
>   call DMGetCoordinateDM(SolScal,CoordDM,ierr)
>   call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)
>
>   write(*,*) coords(:,0,0,0)
>   write(*,*) coords(:,imax-1,jmax-1,kmax-1)
> gives
>   0.1  0.0  0.0
>   1.0  1.2  1.0
>
> Perfect. Not very logical though, that we need this new DM, so perhaps
> add an example/something in the manual?
>

A DM describes a data layout over a topology. Here is the topology is a
Cartesian grid, and 'dm' describes the data layout of your solution vector.
If you want the layout of the coordinate vector then you need another
DM, namely 'coordDM'.

   Matt


> I'm by the way planning to send you a Fortran example solving van der
> Pol with a 3D DMDA (if you want it). I could include the above code in
> that example if desired.  The example is written with some utility
> functions (discussed in emails a few weeks ago) such that it is easy to
> reuse 95% of a legacy code solving van der Pol (or other PDE). The main
> motivation for adding this example (IMO) is that none of the current DM
> examples show how DMDAs are used for solving a PDE in a very simple
> case, particularly not for Fortran.
>
> One question in this regard: can you have examples split into two files?
> That makes it easier to use Fortran modules in this case.
>
>
> ?smund
>
>
> On 21. okt. 2013 13:55, Jed Brown wrote:
> > ?smund Ervik <asmund.ervik at ntnu.no> writes:
> >
> >> Dear all,
> >>
> >> I'm using a DMDA to manage my grid, and I want to have uniform
> >> coordinates, so I use DMDASetUniformCoordinates. Since I let PETSc
> >> decide the grid partitioning, I need to get the local coordinates, i.e.
> >> xmin,xmax for the local grid. I found the function
> >> DMGetCoordinatesLocal, but I can't seem to figure out how it works (not
> >> in the manual, examples are confusing). What I do is
> >>
> >>   call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
> >>   call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
> >>   call DMDAVecGetArrayF90(SolScal,coordVec,coords,ierr)
> >>
> >> Then I have the "coords" F90 array, which has the same dimensions as my
> >> local part of the DA, i.e. dof:imax:jmax:kmax (all starting at 0).
> >
> > Actually, the dimensions are dim:imax:jmax:kmax.  Use
> >
> >   call DMGetCoordinateDM(SolScal,CoordDM,ierr)
> >   call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)
> >
> >> Where in this array can I find the values xmin,xmax etc? It looks
> >> really strange, e.g. a slice in i-direction gives 0.10 0.00 1.20 0.55
> >> 0.00
> >>
> >> I do realize I can compute these on my own, but I think using
> >> DMGetCoordinatesLocal has less potential for errors.
> >>
> >> Best regards,
> >> ?smund Ervik
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/f30bbce6/attachment-0001.html>

From jedbrown at mcs.anl.gov  Mon Oct 21 07:42:11 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 Oct 2013 07:42:11 -0500
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <52651ABC.1060306@ntnu.no>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no>
Message-ID: <87ppqyztgc.fsf@mcs.anl.gov>

?smund Ervik <asmund.ervik at ntnu.no> writes:

> Thanks Jed,
>
> The following works fine (for future reference):
>
>   call DMDASetUniformCoordinates(SolScal,0.1,1.0,0.0,1.2,0.0,1.0,ierr)
>
>   call DMGetCoordinatesLocal(SolScal,coordVec,ierr)
>   call DMGetCoordinateDM(SolScal,CoordDM,ierr)
>   call DMDAVecGetArrayF90(CoordDM,coordVec,coords,ierr)
>
>   write(*,*) coords(:,0,0,0)
>   write(*,*) coords(:,imax-1,jmax-1,kmax-1)
> gives
>   0.1  0.0  0.0
>   1.0  1.2  1.0
>
> Perfect. Not very logical though, that we need this new DM, so perhaps
> add an example/something in the manual?

The coordinates live in a different space than the DM.  Why is it
illogical to use the coordinate DM to access coordinate vectors?  It is
used in several examples, though none of them in Fortran.

> I'm by the way planning to send you a Fortran example solving van der
> Pol with a 3D DMDA (if you want it). 

"van der Pol" normally refers to a 2-variable ODE system (e.g.,
src/ts/examples/tutorials/ex16.c).  I assume you are talking about a
PDE, however.

> I could include the above code in that example if desired.  The
> example is written with some utility functions (discussed in emails a
> few weeks ago) such that it is easy to reuse 95% of a legacy code
> solving van der Pol (or other PDE). The main motivation for adding
> this example (IMO) is that none of the current DM examples show how
> DMDAs are used for solving a PDE in a very simple case, particularly
> not for Fortran.

We would welcome more Fortran examples.  The best way to contribute is
to add your example in a Git branch and send a pull request or patch.

https://bitbucket.org/petsc/petsc/wiki/Home

> One question in this regard: can you have examples split into two files?
> That makes it easier to use Fortran modules in this case.

You can, though we usually do single-file because it's simpler for
people to copy out and modify.  See
src/vec/vec/examples/tutorials/makefile (ex21f90) for an example using
multiple source files.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/7b47fd2a/attachment.pgp>

From asmund.ervik at ntnu.no  Mon Oct 21 08:05:22 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Mon, 21 Oct 2013 15:05:22 +0200
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <87ppqyztgc.fsf@mcs.anl.gov>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no> <87ppqyztgc.fsf@mcs.anl.gov>
Message-ID: <52652692.7000104@ntnu.no>

On 21. okt. 2013 14:42, Jed Brown wrote:
> 
> The coordinates live in a different space than the DM.  Why is it
> illogical to use the coordinate DM to access coordinate vectors?  It is
> used in several examples, though none of them in Fortran.
> 

I guess it is logical for e.g. unstructured meshes or finite elements,
but for someone like me who just wants 2/4/9 real numbers, it seems a
bit overkill to have another DM.

> 
> "van der Pol" normally refers to a 2-variable ODE system (e.g.,
> src/ts/examples/tutorials/ex16.c).  I assume you are talking about a
> PDE, however.

I am actually talking about that 2-variable ODE. We use it as a test
problem in several of our codes, since it is stiff and has some
"structure", but is much simpler than Euler or Navier-Stokes. In my
example I "distribute" it so that we solve the same equation at all grid
points on all processors, using the same code as a PDE solver but with a
different RHS.

I don't know whether the pattern we use is common or not, so I'll submit
a pull request and then you can see if you think this is useful for
others. Another option would be an example solving full-blown
Navier-Stokes (single phase), but that would take some more work, and I
don't know when I'll have time for that.

?smund

From jedbrown at mcs.anl.gov  Mon Oct 21 08:13:40 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 Oct 2013 08:13:40 -0500
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <52652692.7000104@ntnu.no>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no> <87ppqyztgc.fsf@mcs.anl.gov>
	<52652692.7000104@ntnu.no>
Message-ID: <87hacazrzv.fsf@mcs.anl.gov>

?smund Ervik <asmund.ervik at ntnu.no> writes:

> On 21. okt. 2013 14:42, Jed Brown wrote:
>> 
>> The coordinates live in a different space than the DM.  Why is it
>> illogical to use the coordinate DM to access coordinate vectors?  It is
>> used in several examples, though none of them in Fortran.
>> 
>
> I guess it is logical for e.g. unstructured meshes or finite elements,
> but for someone like me who just wants 2/4/9 real numbers, it seems a
> bit overkill to have another DM.

It would be reasonable to offer DMDAGetUniformCoordinates() and
DMDAGetUniformCoordinatesLocal() that would return your few numbers.

> I am actually talking about that 2-variable ODE. We use it as a test
> problem in several of our codes, since it is stiff and has some
> "structure", but is much simpler than Euler or Navier-Stokes. In my
> example I "distribute" it so that we solve the same equation at all grid
> points on all processors, using the same code as a PDE solver but with a
> different RHS.

Okay, but this has no spatial derivatives?

> I don't know whether the pattern we use is common or not, so I'll submit
> a pull request and then you can see if you think this is useful for
> others. Another option would be an example solving full-blown
> Navier-Stokes (single phase), but that would take some more work, and I
> don't know when I'll have time for that.

Whatever you like.  Note that there are already some related examples in
src/ts/examples/tutorials/advection-diffusion-reaction/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/a1b9e116/attachment.pgp>

From asmund.ervik at ntnu.no  Mon Oct 21 08:25:29 2013
From: asmund.ervik at ntnu.no (=?UTF-8?B?w4VzbXVuZCBFcnZpaw==?=)
Date: Mon, 21 Oct 2013 15:25:29 +0200
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <87hacazrzv.fsf@mcs.anl.gov>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no> <87ppqyztgc.fsf@mcs.anl.gov>
	<52652692.7000104@ntnu.no> <87hacazrzv.fsf@mcs.anl.gov>
Message-ID: <52652B49.2020709@ntnu.no>



On 21. okt. 2013 15:13, Jed Brown wrote:
> 
> Okay, but this has no spatial derivatives?

Correct. But the code is written such that (in our framework) a legacy
serial code which has spatial derivatives in the RHS can be plugged
straight in. I don't have any code that I can share with the world that
includes spatial derivatives in the RHS though. I guess the heat
equation would be fairly straight-forward.

> 
> Whatever you like.  Note that there are already some related examples in
> src/ts/examples/tutorials/advection-diffusion-reaction/
> 

Yes, I've seen them, but since they include TS stuff like IMEX etc.,
they're a bit more complicated for a PETSc noobie. It's sometimes hard
to know which bits are TS specific and which are DM specific (although I
must say that PETSc has remarkably good organization and "namespacing").
That's why I wrote the example for my own sake and for my colleagues.

From jedbrown at mcs.anl.gov  Mon Oct 21 08:34:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 Oct 2013 08:34:09 -0500
Subject: [petsc-users] Getting DMDA local coordinates
In-Reply-To: <52652B49.2020709@ntnu.no>
References: <52651296.6030102@ntnu.no> <87sivuzvlw.fsf@mcs.anl.gov>
	<52651ABC.1060306@ntnu.no> <87ppqyztgc.fsf@mcs.anl.gov>
	<52652692.7000104@ntnu.no> <87hacazrzv.fsf@mcs.anl.gov>
	<52652B49.2020709@ntnu.no>
Message-ID: <CAM9tzSkRmpefOnVBmivzVsa-6_puc3nTrx=+fn7JxqKwoFeiEQ@mail.gmail.com>

Great. Looking forward to your example.
On Oct 21, 2013 8:25 AM, "?smund Ervik" <asmund.ervik at ntnu.no> wrote:

>
>
> On 21. okt. 2013 15:13, Jed Brown wrote:
> >
> > Okay, but this has no spatial derivatives?
>
> Correct. But the code is written such that (in our framework) a legacy
> serial code which has spatial derivatives in the RHS can be plugged
> straight in. I don't have any code that I can share with the world that
> includes spatial derivatives in the RHS though. I guess the heat
> equation would be fairly straight-forward.
>
> >
> > Whatever you like.  Note that there are already some related examples in
> > src/ts/examples/tutorials/advection-diffusion-reaction/
> >
>
> Yes, I've seen them, but since they include TS stuff like IMEX etc.,
> they're a bit more complicated for a PETSc noobie. It's sometimes hard
> to know which bits are TS specific and which are DM specific (although I
> must say that PETSc has remarkably good organization and "namespacing").
> That's why I wrote the example for my own sake and for my colleagues.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/791cb018/attachment.html>

From paulhuaizhang at gmail.com  Mon Oct 21 10:30:28 2013
From: paulhuaizhang at gmail.com (paul zhang)
Date: Mon, 21 Oct 2013 11:30:28 -0400
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
Message-ID: <CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>

Hi Jed,

I have a question for you.

I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
system. Here is what I am doing. I cut my computation domain into many
pieces, in each of them I compute independently by solving fluid equations.
This has nothing to do with PETSc. Finally, I collect all of the
information and load it to a whole A matrix. Then I call PETSc functions
and they will solve this system in parallel. Later, I get the update of x,
add it to the initial guess, and do another iteration.

My question is how PETSc functions work in parallel in my case. There are
two guesses to me. First, PETSc solves its own matrix for each domain using
local processor, although A is a global. For the values like number of
iterations, solution vector, their numbers should have equaled to the
number of processors I applied, but I get only one value for each of them.
The reason is that the processors must talk with each other once all of
their work is done, that is why I received the "all reduced" value. This is
more logical than my second guess.

In the second one, the system is solved in parallel too. But PETSc function
redistributes the global sparse matrix A to each of the processors after
its load is complete. That is to say now each processor may not solve the
its own partition matrix.

Which one is right?

Thanks,
Paul


















-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/46b721a0/attachment-0001.html>

From jedbrown at mcs.anl.gov  Mon Oct 21 10:42:30 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 Oct 2013 10:42:30 -0500
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
Message-ID: <8738nuzl3t.fsf@mcs.anl.gov>

paul zhang <paulhuaizhang at gmail.com> writes:

> I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
> system. Here is what I am doing. I cut my computation domain into many
> pieces, in each of them I compute independently by solving fluid equations.
> This has nothing to do with PETSc. Finally, I collect all of the
> information and load it to a whole A matrix. 

I hope you build parts of this matrix on each processor, as is done in
the examples.  Note the range Istart to Iend here:

http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html

> My question is how PETSc functions work in parallel in my case. There are
> two guesses to me. First, PETSc solves its own matrix for each domain using
> local processor, although A is a global. For the values like number of
> iterations, solution vector, their numbers should have equaled to the
> number of processors I applied, but I get only one value for each of them.
> The reason is that the processors must talk with each other once all of
> their work is done, that is why I received the "all reduced" value. This is
> more logical than my second guess.

It does not work because the solution operators are global, so to solve
the problem, the iteration must be global.

> In the second one, the system is solved in parallel too. But PETSc function
> redistributes the global sparse matrix A to each of the processors after
> its load is complete. That is to say now each processor may not solve the
> its own partition matrix.

Hopefully you build the matrix already-distributed.  The default
_preconditioner_ is local, but the iteration is global.  PETSc does not
"redistribute" the matrix automatically, though if you call
MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
choose them.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/75dd3250/attachment.pgp>

From paulhuaizhang at gmail.com  Mon Oct 21 11:23:11 2013
From: paulhuaizhang at gmail.com (paul zhang)
Date: Mon, 21 Oct 2013 12:23:11 -0400
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <8738nuzl3t.fsf@mcs.anl.gov>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
Message-ID: <CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>

Hi Jed,

Thanks a lot for your answer. It really helps. I built parts of the matrix
on each processor, then collected them into a global one according to their
global position. Actually I used two MPI function instead of the one in the
example, where the local size, as well as the global size is given.
VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?

My continuing question is since the iteration for the system is global. Is
it more efficient if I solve locally instead. ie. solve parts on each of
the processor instead of doing globally.


Thanks again,

Paul







On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> paul zhang <paulhuaizhang at gmail.com> writes:
>
> > I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
> > system. Here is what I am doing. I cut my computation domain into many
> > pieces, in each of them I compute independently by solving fluid
> equations.
> > This has nothing to do with PETSc. Finally, I collect all of the
> > information and load it to a whole A matrix.
>
> I hope you build parts of this matrix on each processor, as is done in
> the examples.  Note the range Istart to Iend here:
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>
> > My question is how PETSc functions work in parallel in my case. There are
> > two guesses to me. First, PETSc solves its own matrix for each domain
> using
> > local processor, although A is a global. For the values like number of
> > iterations, solution vector, their numbers should have equaled to the
> > number of processors I applied, but I get only one value for each of
> them.
> > The reason is that the processors must talk with each other once all of
> > their work is done, that is why I received the "all reduced" value. This
> is
> > more logical than my second guess.
>
> It does not work because the solution operators are global, so to solve
> the problem, the iteration must be global.
>
> > In the second one, the system is solved in parallel too. But PETSc
> function
> > redistributes the global sparse matrix A to each of the processors after
> > its load is complete. That is to say now each processor may not solve the
> > its own partition matrix.
>
> Hopefully you build the matrix already-distributed.  The default
> _preconditioner_ is local, but the iteration is global.  PETSc does not
> "redistribute" the matrix automatically, though if you call
> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
> choose them.
>



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/138ba678/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct 21 11:32:53 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 11:32:53 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <3178358.Nb05eOjZJN@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071>
Message-ID: <8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>


   The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.

   BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3

   In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.


   Barry



On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:

> Dear list members,
> 
> I have noticed strange memory consumption after upgrading to 3.4 series. I 
> never had time to properly investigate, but here is what happens [yes, this 
> might be a petsc4py issue, but I doubt it] is
> 
> # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
> # usage data from /proc
> import helpers
> procdata=helpers._ProcessMemoryInfoProc()
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> from petsc4py import PETSc
> procdata=helpers._ProcessMemoryInfoProc()
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> da = PETSc.DA().create(sizes=[100,100,100],
>                       proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
>                       boundary_type=[3,0,0],
>                       stencil_type=PETSc.DA.StencilType.BOX,
>                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> procdata=helpers._ProcessMemoryInfoProc()
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> vec=da.createGlobalVec()
> procdata=helpers._ProcessMemoryInfoProc()
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> 
> outputs
> 
> 48 MiB / 49348 kB
> 48 MiB / 49360 kB
> 381 MiB / 446228 kB
> 435 MiB / 446228 kB
> 
> Which is odd: size of the actual data to be stored in the da is just about 56 
> megabytes, so why does creating the da consume 7 times that? And why does the 
> DA reserve the memory in the first place? I thought memory only gets allocated 
> once an associated vector is created and it indeed looks like the 
> createGlobalVec call does indeed allocate the right amount of data. But what 
> is that 330 MiB that DA().create() consumes? [It's actually the .setUp() 
> method that does the consuming, but that's not of much use as it needs to be 
> called before a vector can be created.]
> 
> Cheers,
> Juha
> 


From knepley at gmail.com  Mon Oct 21 11:52:13 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 11:52:13 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
Message-ID: <CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>

On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    The PETSc DMDA object greedily allocates several arrays of data used to
> set up the communication and other things like local to global mappings
> even before you create any vectors. This is why you see this big bump in
> memory usage.
>
>    BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier;
> at least we did not intend to make it worse. Are you sure it is using more
> memory than in 3.3
>
>    In order for use to decrease the memory usage of the DMDA setup it
> would be helpful if we knew which objects created within it used the most
> memory.  There is some sloppiness in that routine of not reusing memory as
> well as could be, not sure how much difference that would make.


I am adding a DMDA example to look at this is detail. Here is what I have
up front. Suppose that there are G grid vertices, e,g, 10^6 in
your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA
allocates

  Create ltog scatter          dof*8G
  Create gtol scatter          dof*8G
  Raw indices                    dof*4G
  Create ltogmap               dof*4G
  Create ltogmapb                   4G
--------------------------------------------
                                            dof*24G + 4G < 4 vectors

It also allocates 2 temporary vectors which are freed but your test may
pick up since the OS might not have garbage collected them. I will
get the precise numbers for 3D, but they should be similar.

I don't really see the point of using a DMDA without the scatters. You
could save 1 vector of storage by making the creation of the l2g maps
for the global vector lazy (and possibly those indices we use to remap
arrays).

   Matt


>    Barry
>
>
>
> On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
>
> > Dear list members,
> >
> > I have noticed strange memory consumption after upgrading to 3.4 series.
> I
> > never had time to properly investigate, but here is what happens [yes,
> this
> > might be a petsc4py issue, but I doubt it] is
> >
> > # helpers contains _ProcessMemoryInfoProc routine which just digs the
> memory
> > # usage data from /proc
> > import helpers
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > from petsc4py import PETSc
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > da = PETSc.DA().create(sizes=[100,100,100],
> >
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >                       boundary_type=[3,0,0],
> >                       stencil_type=PETSc.DA.StencilType.BOX,
> >                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > vec=da.createGlobalVec()
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >
> > outputs
> >
> > 48 MiB / 49348 kB
> > 48 MiB / 49360 kB
> > 381 MiB / 446228 kB
> > 435 MiB / 446228 kB
> >
> > Which is odd: size of the actual data to be stored in the da is just
> about 56
> > megabytes, so why does creating the da consume 7 times that? And why
> does the
> > DA reserve the memory in the first place? I thought memory only gets
> allocated
> > once an associated vector is created and it indeed looks like the
> > createGlobalVec call does indeed allocate the right amount of data. But
> what
> > is that 330 MiB that DA().create() consumes? [It's actually the .setUp()
> > method that does the consuming, but that's not of much use as it needs
> to be
> > called before a vector can be created.]
> >
> > Cheers,
> > Juha
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/dc6abf47/attachment.html>

From knepley at gmail.com  Mon Oct 21 11:53:15 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 11:53:15 -0500
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
Message-ID: <CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>

On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:

> Hi Jed,
>
> Thanks a lot for your answer. It really helps. I built parts of the matrix
> on each processor, then collected them into a global one according to their
> global position. Actually I used two MPI function instead of the one in the
> example, where the local size, as well as the global size is given.
> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>
> My continuing question is since the iteration for the system is global. Is
> it more efficient if I solve locally instead. ie. solve parts on each of
> the processor instead of doing globally.
>

No, because this ignores the coupling between domains.

  Matt


> Thanks again,
>
> Paul
>
>
>
>
>
>
>
> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> paul zhang <paulhuaizhang at gmail.com> writes:
>>
>> > I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
>> > system. Here is what I am doing. I cut my computation domain into many
>> > pieces, in each of them I compute independently by solving fluid
>> equations.
>> > This has nothing to do with PETSc. Finally, I collect all of the
>> > information and load it to a whole A matrix.
>>
>> I hope you build parts of this matrix on each processor, as is done in
>> the examples.  Note the range Istart to Iend here:
>>
>>
>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>
>> > My question is how PETSc functions work in parallel in my case. There
>> are
>> > two guesses to me. First, PETSc solves its own matrix for each domain
>> using
>> > local processor, although A is a global. For the values like number of
>> > iterations, solution vector, their numbers should have equaled to the
>> > number of processors I applied, but I get only one value for each of
>> them.
>> > The reason is that the processors must talk with each other once all of
>> > their work is done, that is why I received the "all reduced" value.
>> This is
>> > more logical than my second guess.
>>
>> It does not work because the solution operators are global, so to solve
>> the problem, the iteration must be global.
>>
>> > In the second one, the system is solved in parallel too. But PETSc
>> function
>> > redistributes the global sparse matrix A to each of the processors after
>> > its load is complete. That is to say now each processor may not solve
>> the
>> > its own partition matrix.
>>
>> Hopefully you build the matrix already-distributed.  The default
>> _preconditioner_ is local, but the iteration is global.  PETSc does not
>> "redistribute" the matrix automatically, though if you call
>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>> choose them.
>>
>
>
>
> --
> Huaibao (Paul) Zhang
> *Gas Surface Interactions Lab*
> Department of Mechanical Engineering
> University of Kentucky,
> Lexington,
> KY, 40506-0503*
> Office*: 216 Ralph G. Anderson Building
> *Web*:gsil.engineering.uky.edu
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/4fabb718/attachment-0001.html>

From fd.kong at siat.ac.cn  Mon Oct 21 12:26:41 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Mon, 21 Oct 2013 11:26:41 -0600
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
	<DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
Message-ID: <CAN5Wd-L7dK4jPj_Tj40=pLwwx-EwcztWxBWC5_vJS3YEtKPCrA@mail.gmail.com>

Hi Barry,

I contacted the supercomputer center, and they asked me for a test case so
that they can forward it to IBM. Is it possible that we write a test case
only using MPI?  It is not a good idea that we send them the whole petsc
code and my code.


On Sun, Oct 20, 2013 at 2:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   It is unfortunate IBM has perpetuated this error in their libraries and
> made it worse.
>
>    You can, of course, work around it by making your application code far
> more complicated and manage the matrix assembly yourself but that is not a
> good use of your time or anyone's time. Plus how do you know that their
> bugging MPI won't bite you somewhere else, like in the new code you would
> need to write.
>
>    You need to report this bug to IBM and they need to take it seriously,
> unfortunately if you are not the purchaser of the IBM you are running on
> they may not care (companies only care about paying customers who complain).
>
>    Can you just use the Intel MPI mpi compilers/libraries? Or switch to
> some other system that is not from IBM? Better to just not use this machine
> until IBM straightens it out.
>
>    Barry
>
>
>
> On Oct 20, 2013, at 3:04 PM, Fande Kong <fd.kong at siat.ac.cn> wrote:
>
> > This behaviour is really, really strange.
> >
> > The yellowstone supercomputer updated the IBM PE to version 1.3.0.4
> about two months ago (
> https://dailyb.cisl.ucar.edu/bulletins/yellowstone-outage-august-27-update-ibm-pe-and-lsf).
> I recompiled the petsc and my code as they suggested. Unfortunately, this
> problem reoccurs even with small the number of processors (512 cores).
> >
> > The problem only happened before with large the number of processors and
> large the size of problem, but now this problem occurs even with small the
> number of processors and small problem for using IBM MPI or intel MPI.
> >
> > The exactly same code can run on another supercomputer. I think the code
> matstash.c is really sensitive on the IBM PE. It is hard for me to fix it.
> Can we disable stash and then I send off-processors data by ourselves? Or
> can we attach a scatter to mat to exchange off-processors values?
> >
> > error messages:
> >
> > [76]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [76]PETSC ERROR: Petsc has generated inconsistent data!
> > [76]PETSC ERROR: Negative MPI source: stash->nrecvs=5 i=7
> MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=371173!
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: Petsc Release Version 3.4.1, unknown
> > [76]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [76]PETSC ERROR: See docs/index.html for manual pages.
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: ./linearElasticity on a arch-linux2-cxx-opt named
> ys0623 by fandek Sat Oct 19 00:26:16 2013
> > [76]PETSC ERROR: Libraries linked from
> /glade/p/work/fandek/petsc/arch-linux2-cxx-opt/lib
> > [76]PETSC ERROR: Configure run at Fri Oct 18 23:57:35 2013
> > [76]PETSC ERROR: Configure options --with-valgrind=1
> --with-clanguage=cxx --with-shared-libraries=1 --with-dynamic-loading=1
> --download-f-blas-lapack=1 --with-mpi=1 --download-parmetis=1
> --download-metis=1 --with-64-bit-indices=1 --download-netcdf=1
> --download-exodusii=1 --download-ptscotch=1 --download-hdf5=1
> --with-debugging=no
> > [76]PETSC ERROR:
> ------------------------------------------------------------------------
> > [76]PETSC ERROR: MatStashScatterGetMesg_Private() line 633 in
> /glade/p/work/fandek/petsc/src/mat/utils/matstash.c
> > [76]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 676 in
> /glade/p/work/fandek/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [76]PETSC ERROR: MatAssemblyEnd() line 4939 in
> /glade/p/work/fandek/petsc/src/mat/interface/matrix.c
> > [76]PETSC ERROR: SpmcsDMMeshCreatVertexMatrix() line 65 in
> meshreorder.cpp
> > [76]PETSC ERROR: SpmcsDMMeshReOrderingMeshPoints() line 125 in
> meshreorder.cpp
> > [76]PETSC ERROR: CreateProblem() line 59 in preProcessSetUp.cpp
> > [76]PETSC ERROR: DMmeshInitialize() line 95 in mgInitialize.cpp
> > [76]PETSC ERROR: main() line 69 in linearElasticity3d.cpp
> > Abort(77) on node 76 (rank 76 in comm 1140850688): application called
> MPI_Abort(MPI_COMM_WORLD, 77) - process 76
> > ERROR: 0031-300  Forcing all remote tasks to exit due to exit code 1 in
> task 76
> >
> >
> > Thanks
> >
> >
> >
> > On Wed, Jun 26, 2013 at 8:56 AM, Jeff Hammond <jhammond at alcf.anl.gov>
> wrote:
> > This concerns IBM PE-MPI on iDataPlex, which is likely based upon the
> cluster implementation of PAMI, which is a completely different code base
> from the PAMI Blue Gene implementation.  If you can reproduce it on Blue
> Gene/Q, I will care.
> >
> > As an IBM customer, NCAR is endowed with the ability to file bug reports
> directly with IBM related to the products they possess.  There is a link to
> their support system on http://www2.cisl.ucar.edu/resources/yellowstone,
> which is the appropriate channel for users of Yellowstone that have issues
> with the system software installed there.
> >
> > Jeff
> >
> > ----- Original Message -----
> > From: "Jed Brown" <jedbrown at mcs.anl.gov>
> > To: "Fande Kong" <fd.kong at siat.ac.cn>, "petsc-users" <
> petsc-users at mcs.anl.gov>
> > Cc: "Jeff Hammond" <jhammond at alcf.anl.gov>
> > Sent: Wednesday, June 26, 2013 9:21:48 AM
> > Subject: Re: [petsc-users] How to understand these error messages
> >
> > Fande Kong <fd.kong at siat.ac.cn> writes:
> >
> > > Hi Barry,
> > >
> > > If I use the intel mpi, my code can correctly run and can produce some
> > > correct results. Yes, you are right. The IBM MPI has some bugs.
> >
> > Fande, please report this issue to the IBM.
> >
> > Jeff, Fande has a reproducible case where when running on 10k cores and
> > problem sizes over 100M, this
> >
> >   MPI_Waitany(2*stash->nrecvs,stash->recv_waits,&i,&recv_status);
> >
> > returns
> >
> >       [6724]PETSC ERROR: Negative MPI source: stash->nrecvs=8 i=11
> >       MPI_SOURCE=-32766 MPI_TAG=-32766 MPI_ERROR=20613892!
> >
> > It runs correctly for smaller problem sizes, smaller core counts, or for
> > all sizes when using Intel MPI.  This is on Yellowstone (iDataPlex, 4500
> > dx360 nodes).  Do you know someone at IBM that should be notified?
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > University of Chicago Computation Institute
> > jhammond at alcf.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> > ALCF docs: http://www.alcf.anl.gov/user-guides
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/487826aa/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct 21 13:00:10 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 13:00:10 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>
Message-ID: <E18F6450-B483-4EA3-A75F-36BD37724A0F@mcs.anl.gov>


  Matt,

   The scatters should always use block indices (I think they do) so the memory usage for the scatters should not have a dof* in front of this. Are you sure that the dof* is there? If it is there is it because it is a block size that we don't support directly?  We currently have special support for BS or 1,2,3,4,5,6,7,8,12    We should at least fill in 9,10,11

    Do we somewhere inside the VecScatter create business mistakenly create an array that depends on dof*?


   Barry




On Oct 21, 2013, at 11:52 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.
> 
>    BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3
> 
>    In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.
> 
> I am adding a DMDA example to look at this is detail. Here is what I have up front. Suppose that there are G grid vertices, e,g, 10^6 in
> your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA allocates
> 
>   Create ltog scatter          dof*8G
>   Create gtol scatter          dof*8G
>   Raw indices                    dof*4G
>   Create ltogmap               dof*4G
>   Create ltogmapb                   4G
> --------------------------------------------
>                                             dof*24G + 4G < 4 vectors
> 
> It also allocates 2 temporary vectors which are freed but your test may pick up since the OS might not have garbage collected them. I will
> get the precise numbers for 3D, but they should be similar.
> 
> I don't really see the point of using a DMDA without the scatters. You could save 1 vector of storage by making the creation of the l2g maps
> for the global vector lazy (and possibly those indices we use to remap arrays).
> 
>    Matt
> 
> 
>    Barry
> 
> 
> 
> On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> 
> > Dear list members,
> >
> > I have noticed strange memory consumption after upgrading to 3.4 series. I
> > never had time to properly investigate, but here is what happens [yes, this
> > might be a petsc4py issue, but I doubt it] is
> >
> > # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
> > # usage data from /proc
> > import helpers
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > from petsc4py import PETSc
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > da = PETSc.DA().create(sizes=[100,100,100],
> >                       proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >                       boundary_type=[3,0,0],
> >                       stencil_type=PETSc.DA.StencilType.BOX,
> >                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > vec=da.createGlobalVec()
> > procdata=helpers._ProcessMemoryInfoProc()
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >
> > outputs
> >
> > 48 MiB / 49348 kB
> > 48 MiB / 49360 kB
> > 381 MiB / 446228 kB
> > 435 MiB / 446228 kB
> >
> > Which is odd: size of the actual data to be stored in the da is just about 56
> > megabytes, so why does creating the da consume 7 times that? And why does the
> > DA reserve the memory in the first place? I thought memory only gets allocated
> > once an associated vector is created and it indeed looks like the
> > createGlobalVec call does indeed allocate the right amount of data. But what
> > is that 330 MiB that DA().create() consumes? [It's actually the .setUp()
> > method that does the consuming, but that's not of much use as it needs to be
> > called before a vector can be created.]
> >
> > Cheers,
> > Juha
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From knepley at gmail.com  Mon Oct 21 13:04:43 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 13:04:43 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <E18F6450-B483-4EA3-A75F-36BD37724A0F@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>
	<E18F6450-B483-4EA3-A75F-36BD37724A0F@mcs.anl.gov>
Message-ID: <CAMYG4GmMUmZqe0_yEEKgYjKiR-fDYRharZUOy4eeiJVR5V90UA@mail.gmail.com>

On Mon, Oct 21, 2013 at 1:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Matt,
>
>    The scatters should always use block indices (I think they do) so the
> memory usage for the scatters should not have a dof* in front of this. Are
> you sure that the dof* is there? If it is there is it because it is a block
> size that we don't support directly?  We currently have special support for
> BS or 1,2,3,4,5,6,7,8,12    We should at least fill in 9,10,11
>
>     Do we somewhere inside the VecScatter create business mistakenly
> create an array that depends on dof*?


I am sure of this dependence. Its very easy to see by just creating the DA
and ending using -malloc_test. If it is intended to use block indices,
this is not happening.

   Matt


>
>    Barry
>
>
>
>
> On Oct 21, 2013, at 11:52 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> > On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> >    The PETSc DMDA object greedily allocates several arrays of data used
> to set up the communication and other things like local to global mappings
> even before you create any vectors. This is why you see this big bump in
> memory usage.
> >
> >    BUT I don't think it should be any worse in 3.4 than in 3.3 or
> earlier; at least we did not intend to make it worse. Are you sure it is
> using more memory than in 3.3
> >
> >    In order for use to decrease the memory usage of the DMDA setup it
> would be helpful if we knew which objects created within it used the most
> memory.  There is some sloppiness in that routine of not reusing memory as
> well as could be, not sure how much difference that would make.
> >
> > I am adding a DMDA example to look at this is detail. Here is what I
> have up front. Suppose that there are G grid vertices, e,g, 10^6 in
> > your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA
> allocates
> >
> >   Create ltog scatter          dof*8G
> >   Create gtol scatter          dof*8G
> >   Raw indices                    dof*4G
> >   Create ltogmap               dof*4G
> >   Create ltogmapb                   4G
> > --------------------------------------------
> >                                             dof*24G + 4G < 4 vectors
> >
> > It also allocates 2 temporary vectors which are freed but your test may
> pick up since the OS might not have garbage collected them. I will
> > get the precise numbers for 3D, but they should be similar.
> >
> > I don't really see the point of using a DMDA without the scatters. You
> could save 1 vector of storage by making the creation of the l2g maps
> > for the global vector lazy (and possibly those indices we use to remap
> arrays).
> >
> >    Matt
> >
> >
> >    Barry
> >
> >
> >
> > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> >
> > > Dear list members,
> > >
> > > I have noticed strange memory consumption after upgrading to 3.4
> series. I
> > > never had time to properly investigate, but here is what happens [yes,
> this
> > > might be a petsc4py issue, but I doubt it] is
> > >
> > > # helpers contains _ProcessMemoryInfoProc routine which just digs the
> memory
> > > # usage data from /proc
> > > import helpers
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > from petsc4py import PETSc
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > da = PETSc.DA().create(sizes=[100,100,100],
> > >
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > >                       boundary_type=[3,0,0],
> > >                       stencil_type=PETSc.DA.StencilType.BOX,
> > >                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > vec=da.createGlobalVec()
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >
> > > outputs
> > >
> > > 48 MiB / 49348 kB
> > > 48 MiB / 49360 kB
> > > 381 MiB / 446228 kB
> > > 435 MiB / 446228 kB
> > >
> > > Which is odd: size of the actual data to be stored in the da is just
> about 56
> > > megabytes, so why does creating the da consume 7 times that? And why
> does the
> > > DA reserve the memory in the first place? I thought memory only gets
> allocated
> > > once an associated vector is created and it indeed looks like the
> > > createGlobalVec call does indeed allocate the right amount of data.
> But what
> > > is that 330 MiB that DA().create() consumes? [It's actually the
> .setUp()
> > > method that does the consuming, but that's not of much use as it needs
> to be
> > > called before a vector can be created.]
> > >
> > > Cheers,
> > > Juha
> > >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/6dc918a6/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Oct 21 13:15:02 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 13:15:02 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <CAMYG4GmMUmZqe0_yEEKgYjKiR-fDYRharZUOy4eeiJVR5V90UA@mail.gmail.com>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>
	<E18F6450-B483-4EA3-A75F-36BD37724A0F@mcs.anl.gov>
	<CAMYG4GmMUmZqe0_yEEKgYjKiR-fDYRharZUOy4eeiJVR5V90UA@mail.gmail.com>
Message-ID: <26A7096C-7D32-4342-9E63-889E33443016@mcs.anl.gov>


   Sorry, my mistake. The double precision work arrays needed inside the VecScatter are dof * number of ghost points  while the space for the indices should be number of local grid points  (Not times dof). Is that what you see? If the space is dof* number of local grid points then something is wrong somewhere along the processing.


   Barry


On Oct 21, 2013, at 1:04 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 21, 2013 at 1:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Matt,
> 
>    The scatters should always use block indices (I think they do) so the memory usage for the scatters should not have a dof* in front of this. Are you sure that the dof* is there? If it is there is it because it is a block size that we don't support directly?  We currently have special support for BS or 1,2,3,4,5,6,7,8,12    We should at least fill in 9,10,11
> 
>     Do we somewhere inside the VecScatter create business mistakenly create an array that depends on dof*?
> 
> I am sure of this dependence. Its very easy to see by just creating the DA and ending using -malloc_test. If it is intended to use block indices,
> this is not happening.
> 
>    Matt
>  
> 
>    Barry
> 
> 
> 
> 
> On Oct 21, 2013, at 11:52 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> > On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.
> >
> >    BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3
> >
> >    In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.
> >
> > I am adding a DMDA example to look at this is detail. Here is what I have up front. Suppose that there are G grid vertices, e,g, 10^6 in
> > your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA allocates
> >
> >   Create ltog scatter          dof*8G
> >   Create gtol scatter          dof*8G
> >   Raw indices                    dof*4G
> >   Create ltogmap               dof*4G
> >   Create ltogmapb                   4G
> > --------------------------------------------
> >                                             dof*24G + 4G < 4 vectors
> >
> > It also allocates 2 temporary vectors which are freed but your test may pick up since the OS might not have garbage collected them. I will
> > get the precise numbers for 3D, but they should be similar.
> >
> > I don't really see the point of using a DMDA without the scatters. You could save 1 vector of storage by making the creation of the l2g maps
> > for the global vector lazy (and possibly those indices we use to remap arrays).
> >
> >    Matt
> >
> >
> >    Barry
> >
> >
> >
> > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> >
> > > Dear list members,
> > >
> > > I have noticed strange memory consumption after upgrading to 3.4 series. I
> > > never had time to properly investigate, but here is what happens [yes, this
> > > might be a petsc4py issue, but I doubt it] is
> > >
> > > # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
> > > # usage data from /proc
> > > import helpers
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > from petsc4py import PETSc
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > da = PETSc.DA().create(sizes=[100,100,100],
> > >                       proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > >                       boundary_type=[3,0,0],
> > >                       stencil_type=PETSc.DA.StencilType.BOX,
> > >                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > vec=da.createGlobalVec()
> > > procdata=helpers._ProcessMemoryInfoProc()
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >
> > > outputs
> > >
> > > 48 MiB / 49348 kB
> > > 48 MiB / 49360 kB
> > > 381 MiB / 446228 kB
> > > 435 MiB / 446228 kB
> > >
> > > Which is odd: size of the actual data to be stored in the da is just about 56
> > > megabytes, so why does creating the da consume 7 times that? And why does the
> > > DA reserve the memory in the first place? I thought memory only gets allocated
> > > once an associated vector is created and it indeed looks like the
> > > createGlobalVec call does indeed allocate the right amount of data. But what
> > > is that 330 MiB that DA().create() consumes? [It's actually the .setUp()
> > > method that does the consuming, but that's not of much use as it needs to be
> > > called before a vector can be created.]
> > >
> > > Cheers,
> > > Juha
> > >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From knepley at gmail.com  Mon Oct 21 13:17:56 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 13:17:56 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <26A7096C-7D32-4342-9E63-889E33443016@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<CAMYG4Gk4=UyH340+n_Y+J-fTjjg+itOVvBTp6vGNdo3AJH12FQ@mail.gmail.com>
	<E18F6450-B483-4EA3-A75F-36BD37724A0F@mcs.anl.gov>
	<CAMYG4GmMUmZqe0_yEEKgYjKiR-fDYRharZUOy4eeiJVR5V90UA@mail.gmail.com>
	<26A7096C-7D32-4342-9E63-889E33443016@mcs.anl.gov>
Message-ID: <CAMYG4GmCQx_e4QM8GY+E-T5LkSpX3JUi=Nn0jygD22OxVP6GaA@mail.gmail.com>

On Mon, Oct 21, 2013 at 1:15 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Sorry, my mistake. The double precision work arrays needed inside the
> VecScatter are dof * number of ghost points  while the space for the
> indices should be number of local grid points  (Not times dof). Is that
> what you see? If the space is dof* number of local grid points then
> something is wrong somewhere along the processing.


No, it looks like dof*local grid points, unless I am making a mistake. Run
this with -malloc test and change dof from 4 to 2. You will see the memory
easily.

  Matt


>
>    Barry
>
>
> On Oct 21, 2013, at 1:04 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> > On Mon, Oct 21, 2013 at 1:00 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Matt,
> >
> >    The scatters should always use block indices (I think they do) so the
> memory usage for the scatters should not have a dof* in front of this. Are
> you sure that the dof* is there? If it is there is it because it is a block
> size that we don't support directly?  We currently have special support for
> BS or 1,2,3,4,5,6,7,8,12    We should at least fill in 9,10,11
> >
> >     Do we somewhere inside the VecScatter create business mistakenly
> create an array that depends on dof*?
> >
> > I am sure of this dependence. Its very easy to see by just creating the
> DA and ending using -malloc_test. If it is intended to use block indices,
> > this is not happening.
> >
> >    Matt
> >
> >
> >    Barry
> >
> >
> >
> >
> > On Oct 21, 2013, at 11:52 AM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > > On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >    The PETSc DMDA object greedily allocates several arrays of data
> used to set up the communication and other things like local to global
> mappings even before you create any vectors. This is why you see this big
> bump in memory usage.
> > >
> > >    BUT I don't think it should be any worse in 3.4 than in 3.3 or
> earlier; at least we did not intend to make it worse. Are you sure it is
> using more memory than in 3.3
> > >
> > >    In order for use to decrease the memory usage of the DMDA setup it
> would be helpful if we knew which objects created within it used the most
> memory.  There is some sloppiness in that routine of not reusing memory as
> well as could be, not sure how much difference that would make.
> > >
> > > I am adding a DMDA example to look at this is detail. Here is what I
> have up front. Suppose that there are G grid vertices, e,g, 10^6 in
> > > your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA
> allocates
> > >
> > >   Create ltog scatter          dof*8G
> > >   Create gtol scatter          dof*8G
> > >   Raw indices                    dof*4G
> > >   Create ltogmap               dof*4G
> > >   Create ltogmapb                   4G
> > > --------------------------------------------
> > >                                             dof*24G + 4G < 4 vectors
> > >
> > > It also allocates 2 temporary vectors which are freed but your test
> may pick up since the OS might not have garbage collected them. I will
> > > get the precise numbers for 3D, but they should be similar.
> > >
> > > I don't really see the point of using a DMDA without the scatters. You
> could save 1 vector of storage by making the creation of the l2g maps
> > > for the global vector lazy (and possibly those indices we use to remap
> arrays).
> > >
> > >    Matt
> > >
> > >
> > >    Barry
> > >
> > >
> > >
> > > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > >
> > > > Dear list members,
> > > >
> > > > I have noticed strange memory consumption after upgrading to 3.4
> series. I
> > > > never had time to properly investigate, but here is what happens
> [yes, this
> > > > might be a petsc4py issue, but I doubt it] is
> > > >
> > > > # helpers contains _ProcessMemoryInfoProc routine which just digs
> the memory
> > > > # usage data from /proc
> > > > import helpers
> > > > procdata=helpers._ProcessMemoryInfoProc()
> > > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > from petsc4py import PETSc
> > > > procdata=helpers._ProcessMemoryInfoProc()
> > > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > da = PETSc.DA().create(sizes=[100,100,100],
> > > >
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > > >                       boundary_type=[3,0,0],
> > > >                       stencil_type=PETSc.DA.StencilType.BOX,
> > > >                       dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > > > procdata=helpers._ProcessMemoryInfoProc()
> > > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > vec=da.createGlobalVec()
> > > > procdata=helpers._ProcessMemoryInfoProc()
> > > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > >
> > > > outputs
> > > >
> > > > 48 MiB / 49348 kB
> > > > 48 MiB / 49360 kB
> > > > 381 MiB / 446228 kB
> > > > 435 MiB / 446228 kB
> > > >
> > > > Which is odd: size of the actual data to be stored in the da is just
> about 56
> > > > megabytes, so why does creating the da consume 7 times that? And why
> does the
> > > > DA reserve the memory in the first place? I thought memory only gets
> allocated
> > > > once an associated vector is created and it indeed looks like the
> > > > createGlobalVec call does indeed allocate the right amount of data.
> But what
> > > > is that 330 MiB that DA().create() consumes? [It's actually the
> .setUp()
> > > > method that does the consuming, but that's not of much use as it
> needs to be
> > > > called before a vector can be created.]
> > > >
> > > > Cheers,
> > > > Juha
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/55e697d6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex1.c
Type: text/x-csrc
Size: 1666 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/55e697d6/attachment.c>

From a.vergottis at ucl.ac.uk  Mon Oct 21 14:31:44 2013
From: a.vergottis at ucl.ac.uk (Anthony Vergottis)
Date: Mon, 21 Oct 2013 20:31:44 +0100
Subject: [petsc-users] Including user made shared library in makefile.
Message-ID: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>

Dear All,

I was wondering how does one include a .so object when compiling with a
PETSc makefile? I have made various .so object for many projects and they
work properly by specifying the -L location and the -lLibrary name. But
when I try to link the .so object via a PETSc make file (after compilation
with no errors) the ldd command does not show the library as being linked.

Is there any internal behaviour of PETSc I should be aware of?  Have tried
everything possible.

I have configure PETSc with the following flags (if this helps).

--download-mpich --with-clanguage=c++ --download-f-blas-lapack
--download-metis --download-parmetis --with-shared-libraries=1
--with-dynamic-loading=1

This is the sample makefile:

include ${PETSC_DIR}/conf/variables
include ${PETSC_DIR}/conf/rules
LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro

pro: pro.o 2D_CBS_Solver.o
            ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
            ${RM} pro.o 2D_CBS_Solver.o

After this makefile is run still ldd does not show linkage.

Any help would be appreciated. Thanks in advance.

Best regards,
Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/87934b27/attachment.html>

From knepley at gmail.com  Mon Oct 21 14:34:35 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 14:34:35 -0500
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
Message-ID: <CAMYG4Gmw6F7uOpxA_ZSFyMCfYtG2FS=MpbD8SonbnF2SkX8sLg@mail.gmail.com>

On Mon, Oct 21, 2013 at 2:31 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk>wrote:

> Dear All,
>
> I was wondering how does one include a .so object when compiling with a
> PETSc makefile? I have made various .so object for many projects and they
> work properly by specifying the -L location and the -lLibrary name. But
> when I try to link the .so object via a PETSc make file (after compilation
> with no errors) the ldd command does not show the library as being linked.
>

If you are in Linux (it looks like you are), then you will need -rpath
along with -L to get the shared library picked up. Look at the PETSc link
line.

   Matt


> Is there any internal behaviour of PETSc I should be aware of?  Have tried
> everything possible.
>
> I have configure PETSc with the following flags (if this helps).
>
> --download-mpich --with-clanguage=c++ --download-f-blas-lapack
> --download-metis --download-parmetis --with-shared-libraries=1
> --with-dynamic-loading=1
>
> This is the sample makefile:
>
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
>
> pro: pro.o 2D_CBS_Solver.o
>             ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
>             ${RM} pro.o 2D_CBS_Solver.o
>
> After this makefile is run still ldd does not show linkage.
>
> Any help would be appreciated. Thanks in advance.
>
> Best regards,
> Anthony
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/612b29b8/attachment-0001.html>

From balay at mcs.anl.gov  Mon Oct 21 14:35:45 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 21 Oct 2013 14:35:45 -0500 (CDT)
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>

On Mon, 21 Oct 2013, Anthony Vergottis wrote:

> Dear All,
> 
> I was wondering how does one include a .so object when compiling with a
> PETSc makefile? I have made various .so object for many projects and they
> work properly by specifying the -L location and the -lLibrary name. But
> when I try to link the .so object via a PETSc make file (after compilation
> with no errors) the ldd command does not show the library as being linked.
> 
> Is there any internal behaviour of PETSc I should be aware of?  Have tried
> everything possible.
> 
> I have configure PETSc with the following flags (if this helps).
> 
> --download-mpich --with-clanguage=c++ --download-f-blas-lapack
> --download-metis --download-parmetis --with-shared-libraries=1
> --with-dynamic-loading=1
> 
> This is the sample makefile:
> 
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro

you need:

LIBS=-L/home/adonis/Dropbox/ParallelSolver -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro

Satish

> 
> pro: pro.o 2D_CBS_Solver.o
>             ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
>             ${RM} pro.o 2D_CBS_Solver.o
> 
> After this makefile is run still ldd does not show linkage.
> 
> Any help would be appreciated. Thanks in advance.
> 
> Best regards,
> Anthony
> 


From a.vergottis at ucl.ac.uk  Mon Oct 21 14:53:02 2013
From: a.vergottis at ucl.ac.uk (Anthony Vergottis)
Date: Mon, 21 Oct 2013 20:53:02 +0100
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
Message-ID: <CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>

Thanks for the replies. Unfortunatly this has still not worked. When using
the ldd command the shared library libpro.so is not listed there. Any other
ideas why this may be not linking?

Thanks again.


On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
>
> > Dear All,
> >
> > I was wondering how does one include a .so object when compiling with a
> > PETSc makefile? I have made various .so object for many projects and they
> > work properly by specifying the -L location and the -lLibrary name. But
> > when I try to link the .so object via a PETSc make file (after
> compilation
> > with no errors) the ldd command does not show the library as being
> linked.
> >
> > Is there any internal behaviour of PETSc I should be aware of?  Have
> tried
> > everything possible.
> >
> > I have configure PETSc with the following flags (if this helps).
> >
> > --download-mpich --with-clanguage=c++ --download-f-blas-lapack
> > --download-metis --download-parmetis --with-shared-libraries=1
> > --with-dynamic-loading=1
> >
> > This is the sample makefile:
> >
> > include ${PETSC_DIR}/conf/variables
> > include ${PETSC_DIR}/conf/rules
> > LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
>
> you need:
>
> LIBS=-L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>
> Satish
>
> >
> > pro: pro.o 2D_CBS_Solver.o
> >             ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
> >             ${RM} pro.o 2D_CBS_Solver.o
> >
> > After this makefile is run still ldd does not show linkage.
> >
> > Any help would be appreciated. Thanks in advance.
> >
> > Best regards,
> > Anthony
> >
>
>
>
>


-- 
Anthony Vergottis

PhD Student
Department of Mechanical Engineering
University College London
Gower Street
London
WC1E 6BT

E-mail: a.vergottis at ucl.ac.uk <i.stojanovic at ucl.ac.uk>
Tel (UK Mobile): (+44) 079 1263 0064
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/43cbd17f/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct 21 14:56:54 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 14:56:54 -0500
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
	<CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
Message-ID: <B6DDBAC3-517E-4426-953E-8577B22D6747@mcs.anl.gov>


  Please cut and paste everything you've done in this process, the make, the running of ldd etc

   Barry

On Oct 21, 2013, at 2:53 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk> wrote:

> Thanks for the replies. Unfortunatly this has still not worked. When using the ldd command the shared library libpro.so is not listed there. Any other ideas why this may be not linking?
> 
> Thanks again.
> 
> 
> On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov> wrote:
> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
> 
> > Dear All,
> >
> > I was wondering how does one include a .so object when compiling with a
> > PETSc makefile? I have made various .so object for many projects and they
> > work properly by specifying the -L location and the -lLibrary name. But
> > when I try to link the .so object via a PETSc make file (after compilation
> > with no errors) the ldd command does not show the library as being linked.
> >
> > Is there any internal behaviour of PETSc I should be aware of?  Have tried
> > everything possible.
> >
> > I have configure PETSc with the following flags (if this helps).
> >
> > --download-mpich --with-clanguage=c++ --download-f-blas-lapack
> > --download-metis --download-parmetis --with-shared-libraries=1
> > --with-dynamic-loading=1
> >
> > This is the sample makefile:
> >
> > include ${PETSC_DIR}/conf/variables
> > include ${PETSC_DIR}/conf/rules
> > LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
> 
> you need:
> 
> LIBS=-L/home/adonis/Dropbox/ParallelSolver -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
> 
> Satish
> 
> >
> > pro: pro.o 2D_CBS_Solver.o
> >             ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
> >             ${RM} pro.o 2D_CBS_Solver.o
> >
> > After this makefile is run still ldd does not show linkage.
> >
> > Any help would be appreciated. Thanks in advance.
> >
> > Best regards,
> > Anthony
> >
> 
> 
> 
> 
> 
> 
> -- 
> Anthony Vergottis
> 
> PhD Student
> Department of Mechanical Engineering
> University College London
> Gower Street
> London 
> WC1E 6BT
> 
> E-mail: a.vergottis at ucl.ac.uk  
> Tel (UK Mobile): (+44) 079 1263 0064


From klaus.zimmermann at physik.uni-freiburg.de  Mon Oct 21 15:06:29 2013
From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann)
Date: Mon, 21 Oct 2013 22:06:29 +0200
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
	<CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
Message-ID: <52658945.9030709@physik.uni-freiburg.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Anthony,

is it actually used, i.e. do you in this test setup call a function
that is implemented in the library?

On a side note: Why do you remove the object files? This seems a bit
unorthodox in a makefile.

Cheers
Klaus

Am 21.10.2013 21:53, schrieb Anthony Vergottis:
> Thanks for the replies. Unfortunatly this has still not worked.
> When using the ldd command the shared library libpro.so is not
> listed there. Any other ideas why this may be not linking?
> 
> Thanks again.
> 
> 
> On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov 
> <mailto:balay at mcs.anl.gov>> wrote:
> 
> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
> 
>> Dear All,
>> 
>> I was wondering how does one include a .so object when compiling
> with a
>> PETSc makefile? I have made various .so object for many projects
> and they
>> work properly by specifying the -L location and the -lLibrary
> name. But
>> when I try to link the .so object via a PETSc make file (after
> compilation
>> with no errors) the ldd command does not show the library as
>> being
> linked.
>> 
>> Is there any internal behaviour of PETSc I should be aware of?
> Have tried
>> everything possible.
>> 
>> I have configure PETSc with the following flags (if this helps).
>> 
>> --download-mpich --with-clanguage=c++ --download-f-blas-lapack 
>> --download-metis --download-parmetis --with-shared-libraries=1 
>> --with-dynamic-loading=1
>> 
>> This is the sample makefile:
>> 
>> include ${PETSC_DIR}/conf/variables include
>> ${PETSC_DIR}/conf/rules 
>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
> 
> you need:
> 
> LIBS=-L/home/adonis/Dropbox/ParallelSolver 
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
> 
> Satish
> 
>> 
>> pro: pro.o 2D_CBS_Solver.o ${CLINKER} -o pro pro.o
>> 2D_CBS_Solver.o ${PETSC_LIB}
> ${LIBS}
>> ${RM} pro.o 2D_CBS_Solver.o
>> 
>> After this makefile is run still ldd does not show linkage.
>> 
>> Any help would be appreciated. Thanks in advance.
>> 
>> Best regards, Anthony
>> 
> 
> 
> 
> 
> 
> 
> -- Anthony Vergottis
> 
> PhD Student Department of Mechanical Engineering University College
> London Gower Street London WC1E 6BT
> 
> E-mail: a.vergottis at ucl.ac.uk <mailto:i.stojanovic at ucl.ac.uk> Tel
> (UK Mobile): (+44) 079 1263 0064

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSZYlFAAoJEHGtO1AgqpgxI+wH/3ItAioJSqUoZKQt5uwTbxlI
SQqtYtwgyw8Xt6PvK6Nua5lPz7wbWTlTl+O3B6u/qqvLjFbQzVF+lVgIlxlil0Ik
5o9pSPuiSjHP06msZ7bDZUeErAX8X2y6q9xlo0QiVfmDGoQgZnarpjdHg8/yzfBW
hGNkGU8pUlVkIGqXObIJiElUw8PnFRG+c/OzOnMZV5Eoa67njBY2tXhW5FmgRszc
0nbMZa6tgiRvhxgEcTS+oxw+l1RT+opebUDIX3v5bKPhiSwAgrtfrQ4yhQi2oceZ
bVSUD2fZFAGbkpJJLESyznABlp/5ffE24RxvmHcNEmXICjvc4vJ7IAigYjdg7Os=
=xt5K
-----END PGP SIGNATURE-----

From bsmith at mcs.anl.gov  Mon Oct 21 15:19:48 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 15:19:48 -0500
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CD90321D-D222-41BB-9948-15D539768634@mcs.anl.gov>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
	<CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
	<B6DDBAC3-517E-4426-953E-8577B22D6747@mcs.anl.gov>
	<CACH9r5KXw-4nUmyovfBy10wDvrQaDbwBjHngZYTMNt80NmC0_g@mail.gmail.com>
	<CD90321D-D222-41BB-9948-15D539768634@mcs.anl.gov>
Message-ID: <659EC188-B7DB-4DB7-AD77-0E882B5EF206@mcs.anl.gov>


   Please always make sure to include the mailing list in your list of emails. Otherwise it just goes to me, and I don't know what I am talking about.


   Barry

On Oct 21, 2013, at 3:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   What happens when you run 
> 
> ./pro   
> 
> ?
> 
>  Try putting ${LIBS}  before the ${PETSC_LIB}  in the makefile
> 
>   Barry
> 
> 
> On Oct 21, 2013, at 3:10 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk> wrote:
> 
>> OK this is the makefile,
>> 
>> include ${PETSC_DIR}/conf/variables
>> include ${PETSC_DIR}/conf/rules
>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>> 
>> 
>> pro: pro.o 2D_CBS_Solver.o
>>            ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS} 
>>            ${RM} pro.o 2D_CBS_Solver.o
>> 
>> 
>> The terminal output (in bold are my commands),
>> 
>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ g++ -shared -fPIC -o libpro.so 2D_CBS_Solver.cpp
>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ls -l
>> total 84
>> -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:56 2D_CBS_Solver.cpp
>> -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:50 2D_CBS_Solver.cpp~
>> -rw-rw-r-- 1 adonis adonis   429 Oct 21 19:13 2D_CBS_Solver.h
>> drwxrwxr-x 2 adonis adonis  4096 Oct 18 14:31 include
>> -rwxrwxr-x 1 adonis adonis  7968 Oct 21 21:02 libpro.so
>> -rw------- 1 adonis adonis   291 Oct 21 21:01 Makefile
>> -rw------- 1 adonis adonis   291 Oct 21 21:01 Makefile~
>> -rwxrwxr-x 1 adonis adonis 42896 Oct 21 21:02 pro
>> -rw-rw-r-- 1 adonis adonis   940 Oct 21 20:42 pro.cpp
>> drwxrwxr-x 2 adonis adonis  4096 Oct 21 20:54 Solvers
>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ file libpro.so 
>> libpro.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x928d227f5816b3de3329f4364a688b6146df8c60, not stripped
>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ make pro
>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -o pro.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g  -fPIC    -I/home/adonis/Documents/petsc-3.4.2/include -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include    -D__INSDIR__= pro.cpp
>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -o 2D_CBS_Solver.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g  -fPIC    -I/home/adonis/Documents/petsc-3.4.2/include -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include    -D__INSDIR__= 2D_CBS_Solver.cpp
>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g  -o pro pro.o 2D_CBS_Solver.o -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib -L/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib  -lpetsc -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib -lflapack -lfblas -lX11 -lpthread -lparmetis -lmetis -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl  -L/home/adonis/Dropbox/ParallelSolver -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro 
>> /bin/rm -f pro.o 2D_CBS_Solver.o
>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ldd pro
>>    linux-vdso.so.1 =>  (0x00007fff1dffe000)
>>    libpetsc.so => /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libpetsc.so (0x00007f24f4da6000)
>>    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f24f4b88000)
>>    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f24f4888000)
>>    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f24f4672000)
>>    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f24f42b2000)
>>    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f24f3f7d000)
>>    libparmetis.so => /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libparmetis.so (0x00007f24f3d2f000)
>>    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f24f3a17000)
>>    libmpich.so.10 => /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpich.so.10 (0x00007f24f364b000)
>>    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f24f3447000)
>>    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f24f314a000)
>>    /lib64/ld-linux-x86-64.so.2 (0x00007f24f6176000)
>>    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f24f2f2c000)
>>    libmetis.so => /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmetis.so (0x00007f24f2cb0000)
>>    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f24f2a7a000)
>>    libmpl.so.1 => /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpl.so.1 (0x00007f24f2875000)
>>    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f24f266c000)
>>    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f24f2469000)
>>    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f24f2263000)
>> 
>> 
>> I hope this helps and is more clear. The  libpro.so object for some reason is not being linked.
>> 
>> Thanks
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On 21 October 2013 20:56, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>  Please cut and paste everything you've done in this process, the make, the running of ldd etc
>> 
>>   Barry
>> 
>> On Oct 21, 2013, at 2:53 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk> wrote:
>> 
>>> Thanks for the replies. Unfortunatly this has still not worked. When using the ldd command the shared library libpro.so is not listed there. Any other ideas why this may be not linking?
>>> 
>>> Thanks again.
>>> 
>>> 
>>> On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov> wrote:
>>> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
>>> 
>>>> Dear All,
>>>> 
>>>> I was wondering how does one include a .so object when compiling with a
>>>> PETSc makefile? I have made various .so object for many projects and they
>>>> work properly by specifying the -L location and the -lLibrary name. But
>>>> when I try to link the .so object via a PETSc make file (after compilation
>>>> with no errors) the ldd command does not show the library as being linked.
>>>> 
>>>> Is there any internal behaviour of PETSc I should be aware of?  Have tried
>>>> everything possible.
>>>> 
>>>> I have configure PETSc with the following flags (if this helps).
>>>> 
>>>> --download-mpich --with-clanguage=c++ --download-f-blas-lapack
>>>> --download-metis --download-parmetis --with-shared-libraries=1
>>>> --with-dynamic-loading=1
>>>> 
>>>> This is the sample makefile:
>>>> 
>>>> include ${PETSC_DIR}/conf/variables
>>>> include ${PETSC_DIR}/conf/rules
>>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
>>> 
>>> you need:
>>> 
>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>>> 
>>> Satish
>>> 
>>>> 
>>>> pro: pro.o 2D_CBS_Solver.o
>>>>            ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
>>>>            ${RM} pro.o 2D_CBS_Solver.o
>>>> 
>>>> After this makefile is run still ldd does not show linkage.
>>>> 
>>>> Any help would be appreciated. Thanks in advance.
>>>> 
>>>> Best regards,
>>>> Anthony
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Anthony Vergottis
>>> 
>>> PhD Student
>>> Department of Mechanical Engineering
>>> University College London
>>> Gower Street
>>> London
>>> WC1E 6BT
>>> 
>>> E-mail: a.vergottis at ucl.ac.uk
>>> Tel (UK Mobile): (+44) 079 1263 0064
>> 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> Anthony Vergottis
>> 
>> PhD Student
>> Department of Mechanical Engineering
>> University College London
>> Gower Street
>> London 
>> WC1E 6BT
>> 
>> E-mail: a.vergottis at ucl.ac.uk  
>> Tel (UK Mobile): (+44) 079 1263 0064
> 


From bsmith at mcs.anl.gov  Mon Oct 21 15:23:01 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 15:23:01 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
Message-ID: <9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>


   Matt,

     I think you are running on 1 process where the DMDA doesn't have an optimized path, when I run on 2 processes the numbers indicate nothing proportional to dof* number of local points

dof = 12
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 182480 VecScatterCreate_PtoS()

dof = 8
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 176080 VecScatterCreate_PtoS()

dof = 4

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 169680 VecScatterCreate_PtoS()

dof = 2
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 166480 VecScatterCreate_PtoS()

dof =2 grid is 50 by 50 instead of 100 by 100

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 6352 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 43952 VecScatterCreate_PtoS()

The IS creation in the DMDA is far more troubling

/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS

dof = 2

[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 81600 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 42016 ISLocalToGlobalMappingCreate()

dof = 4

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 163200 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 82816 ISLocalToGlobalMappingCreate()

dof = 8

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 326400 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 164416 ISLocalToGlobalMappingCreate()

dof = 12
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 489600 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 246016 ISLocalToGlobalMappingCreate()

Here the accessing of indices is at the point level (as well as block) and hence memory usage is proportional to dof* local number of grid points. Of course it is still only proportional to the vector size. There is some improvement we could make it; with a lot of refactoring we can remove the dof* completely, with a little refactoring we can bring it down to a single dof*local number of grid points.

   I cannot understand why you are seeing memory usage 7 times more than a vector. That seems like a lot.

   Barry



On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.
> 
>   BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3
> 
>   In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.
> 
> 
>   Barry
> 
> 
> 
> On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> 
>> Dear list members,
>> 
>> I have noticed strange memory consumption after upgrading to 3.4 series. I 
>> never had time to properly investigate, but here is what happens [yes, this 
>> might be a petsc4py issue, but I doubt it] is
>> 
>> # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
>> # usage data from /proc
>> import helpers
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> from petsc4py import PETSc
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> da = PETSc.DA().create(sizes=[100,100,100],
>>                      proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
>>                      boundary_type=[3,0,0],
>>                      stencil_type=PETSc.DA.StencilType.BOX,
>>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> vec=da.createGlobalVec()
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> 
>> outputs
>> 
>> 48 MiB / 49348 kB
>> 48 MiB / 49360 kB
>> 381 MiB / 446228 kB
>> 435 MiB / 446228 kB
>> 
>> Which is odd: size of the actual data to be stored in the da is just about 56 
>> megabytes, so why does creating the da consume 7 times that? And why does the 
>> DA reserve the memory in the first place? I thought memory only gets allocated 
>> once an associated vector is created and it indeed looks like the 
>> createGlobalVec call does indeed allocate the right amount of data. But what 
>> is that 330 MiB that DA().create() consumes? [It's actually the .setUp() 
>> method that does the consuming, but that's not of much use as it needs to be 
>> called before a vector can be created.]
>> 
>> Cheers,
>> Juha
>> 
> 


From a.vergottis at ucl.ac.uk  Mon Oct 21 15:33:04 2013
From: a.vergottis at ucl.ac.uk (Anthony Vergottis)
Date: Mon, 21 Oct 2013 21:33:04 +0100
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <659EC188-B7DB-4DB7-AD77-0E882B5EF206@mcs.anl.gov>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>
	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>
	<CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>
	<B6DDBAC3-517E-4426-953E-8577B22D6747@mcs.anl.gov>
	<CACH9r5KXw-4nUmyovfBy10wDvrQaDbwBjHngZYTMNt80NmC0_g@mail.gmail.com>
	<CD90321D-D222-41BB-9948-15D539768634@mcs.anl.gov>
	<659EC188-B7DB-4DB7-AD77-0E882B5EF206@mcs.anl.gov>
Message-ID: <CACH9r5LE6wwihEYRQfUocs0HkpT1fuPT7c7AXN_8hetAN1_hDQ@mail.gmail.com>

When running ./pro the program works as expected, but when the .so file is
recompiled ./pro does not show any change. Change occurs only when the
makefile is rerun. When moving ${LIBS} before ${PETSC_LIB} there is no
change as well.


On 21 October 2013 21:19, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Please always make sure to include the mailing list in your list of
> emails. Otherwise it just goes to me, and I don't know what I am talking
> about.
>
>
>    Barry
>
> On Oct 21, 2013, at 3:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> >
> >   What happens when you run
> >
> > ./pro
> >
> > ?
> >
> >  Try putting ${LIBS}  before the ${PETSC_LIB}  in the makefile
> >
> >   Barry
> >
> >
> > On Oct 21, 2013, at 3:10 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk>
> wrote:
> >
> >> OK this is the makefile,
> >>
> >> include ${PETSC_DIR}/conf/variables
> >> include ${PETSC_DIR}/conf/rules
> >> LIBS=-L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
> >>
> >>
> >> pro: pro.o 2D_CBS_Solver.o
> >>            ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB} ${LIBS}
> >>            ${RM} pro.o 2D_CBS_Solver.o
> >>
> >>
> >> The terminal output (in bold are my commands),
> >>
> >> adonis at LinuxBox:~/Dropbox/ParallelSolver$ g++ -shared -fPIC -o
> libpro.so 2D_CBS_Solver.cpp
> >> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ls -l
> >> total 84
> >> -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:56 2D_CBS_Solver.cpp
> >> -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:50 2D_CBS_Solver.cpp~
> >> -rw-rw-r-- 1 adonis adonis   429 Oct 21 19:13 2D_CBS_Solver.h
> >> drwxrwxr-x 2 adonis adonis  4096 Oct 18 14:31 include
> >> -rwxrwxr-x 1 adonis adonis  7968 Oct 21 21:02 libpro.so
> >> -rw------- 1 adonis adonis   291 Oct 21 21:01 Makefile
> >> -rw------- 1 adonis adonis   291 Oct 21 21:01 Makefile~
> >> -rwxrwxr-x 1 adonis adonis 42896 Oct 21 21:02 pro
> >> -rw-rw-r-- 1 adonis adonis   940 Oct 21 20:42 pro.cpp
> >> drwxrwxr-x 2 adonis adonis  4096 Oct 21 20:54 Solvers
> >> adonis at LinuxBox:~/Dropbox/ParallelSolver$ file libpro.so
> >> libpro.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV),
> dynamically linked,
> BuildID[sha1]=0x928d227f5816b3de3329f4364a688b6146df8c60, not stripped
> >> adonis at LinuxBox:~/Dropbox/ParallelSolver$ make pro
> >> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -o
> pro.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g
>  -fPIC    -I/home/adonis/Documents/petsc-3.4.2/include
> -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include
>  -D__INSDIR__= pro.cpp
> >> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -o
> 2D_CBS_Solver.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -g  -fPIC
>  -I/home/adonis/Documents/petsc-3.4.2/include
> -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include
>  -D__INSDIR__= 2D_CBS_Solver.cpp
> >> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx -Wall
> -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g  -o pro pro.o
> 2D_CBS_Solver.o
> -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib
> -L/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib  -lpetsc
> -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib
> -lflapack -lfblas -lX11 -lpthread -lparmetis -lmetis -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.6
> -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath
> -lm -lmpichcxx -lstdc++ -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s
> -ldl  -L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
> >> /bin/rm -f pro.o 2D_CBS_Solver.o
> >> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ldd pro
> >>    linux-vdso.so.1 =>  (0x00007fff1dffe000)
> >>    libpetsc.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libpetsc.so
> (0x00007f24f4da6000)
> >>    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f24f4b88000)
> >>    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00007f24f4888000)
> >>    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x00007f24f4672000)
> >>    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f24f42b2000)
> >>    libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6
> (0x00007f24f3f7d000)
> >>    libparmetis.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libparmetis.so
> (0x00007f24f3d2f000)
> >>    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3
> (0x00007f24f3a17000)
> >>    libmpich.so.10 =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpich.so.10
> (0x00007f24f364b000)
> >>    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f24f3447000)
> >>    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f24f314a000)
> >>    /lib64/ld-linux-x86-64.so.2 (0x00007f24f6176000)
> >>    libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1
> (0x00007f24f2f2c000)
> >>    libmetis.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmetis.so
> (0x00007f24f2cb0000)
> >>    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
> (0x00007f24f2a7a000)
> >>    libmpl.so.1 =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpl.so.1
> (0x00007f24f2875000)
> >>    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f24f266c000)
> >>    libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6
> (0x00007f24f2469000)
> >>    libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6
> (0x00007f24f2263000)
> >>
> >>
> >> I hope this helps and is more clear. The  libpro.so object for some
> reason is not being linked.
> >>
> >> Thanks
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 21 October 2013 20:56, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>
> >>  Please cut and paste everything you've done in this process, the make,
> the running of ldd etc
> >>
> >>   Barry
> >>
> >> On Oct 21, 2013, at 2:53 PM, Anthony Vergottis <a.vergottis at ucl.ac.uk>
> wrote:
> >>
> >>> Thanks for the replies. Unfortunatly this has still not worked. When
> using the ldd command the shared library libpro.so is not listed there. Any
> other ideas why this may be not linking?
> >>>
> >>> Thanks again.
> >>>
> >>>
> >>> On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov> wrote:
> >>> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
> >>>
> >>>> Dear All,
> >>>>
> >>>> I was wondering how does one include a .so object when compiling with
> a
> >>>> PETSc makefile? I have made various .so object for many projects and
> they
> >>>> work properly by specifying the -L location and the -lLibrary name.
> But
> >>>> when I try to link the .so object via a PETSc make file (after
> compilation
> >>>> with no errors) the ldd command does not show the library as being
> linked.
> >>>>
> >>>> Is there any internal behaviour of PETSc I should be aware of?  Have
> tried
> >>>> everything possible.
> >>>>
> >>>> I have configure PETSc with the following flags (if this helps).
> >>>>
> >>>> --download-mpich --with-clanguage=c++ --download-f-blas-lapack
> >>>> --download-metis --download-parmetis --with-shared-libraries=1
> >>>> --with-dynamic-loading=1
> >>>>
> >>>> This is the sample makefile:
> >>>>
> >>>> include ${PETSC_DIR}/conf/variables
> >>>> include ${PETSC_DIR}/conf/rules
> >>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
> >>>
> >>> you need:
> >>>
> >>> LIBS=-L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
> >>>
> >>> Satish
> >>>
> >>>>
> >>>> pro: pro.o 2D_CBS_Solver.o
> >>>>            ${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB}
> ${LIBS}
> >>>>            ${RM} pro.o 2D_CBS_Solver.o
> >>>>
> >>>> After this makefile is run still ldd does not show linkage.
> >>>>
> >>>> Any help would be appreciated. Thanks in advance.
> >>>>
> >>>> Best regards,
> >>>> Anthony
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Anthony Vergottis
> >>>
> >>> PhD Student
> >>> Department of Mechanical Engineering
> >>> University College London
> >>> Gower Street
> >>> London
> >>> WC1E 6BT
> >>>
> >>> E-mail: a.vergottis at ucl.ac.uk
> >>> Tel (UK Mobile): (+44) 079 1263 0064
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Anthony Vergottis
> >>
> >> PhD Student
> >> Department of Mechanical Engineering
> >> University College London
> >> Gower Street
> >> London
> >> WC1E 6BT
> >>
> >> E-mail: a.vergottis at ucl.ac.uk
> >> Tel (UK Mobile): (+44) 079 1263 0064
> >
>
>
>
>


-- 
Anthony Vergottis

PhD Student
Department of Mechanical Engineering
University College London
Gower Street
London
WC1E 6BT

E-mail: a.vergottis at ucl.ac.uk <i.stojanovic at ucl.ac.uk>
Tel (UK Mobile): (+44) 079 1263 0064
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/d756bee7/attachment.html>

From klaus.zimmermann at physik.uni-freiburg.de  Mon Oct 21 15:41:18 2013
From: klaus.zimmermann at physik.uni-freiburg.de (Klaus Zimmermann)
Date: Mon, 21 Oct 2013 22:41:18 +0200
Subject: [petsc-users] Including user made shared library in makefile.
In-Reply-To: <CACH9r5LE6wwihEYRQfUocs0HkpT1fuPT7c7AXN_8hetAN1_hDQ@mail.gmail.com>
References: <CACH9r5La4xpxhyoXpXbUM_hFAT9rNvy2BZAfVGbumuTyVsWCig@mail.gmail.com>	<alpine.LFD.2.03.1310211434550.2482@mcs.anl.gov>	<CACH9r5KEU_qcjy0EOsnHidBwdYG3tV=g8QH0itFdVnfSGNY_zg@mail.gmail.com>	<B6DDBAC3-517E-4426-953E-8577B22D6747@mcs.anl.gov>	<CACH9r5KXw-4nUmyovfBy10wDvrQaDbwBjHngZYTMNt80NmC0_g@mail.gmail.com>	<CD90321D-D222-41BB-9948-15D539768634@mcs.anl.gov>	<659EC188-B7DB-4DB7-AD77-0E882B5EF206@mcs.anl.gov>
	<CACH9r5LE6wwihEYRQfUocs0HkpT1fuPT7c7AXN_8hetAN1_hDQ@mail.gmail.com>
Message-ID: <5265916E.7020809@physik.uni-freiburg.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Anthony,

the your binary does not use the library. Here is why:

g++ -shared -fPIC -o libpro.so 2D_CBS_Solver.cpp
creates the library with the contents of 2D_CBS_Solver.cpp.
But when you link your executable with
${CLINKER} -o pro pro.o 2D_CBS_Solver.o ${PETSC_LIB}
in your makefile you link 2D_CBS_Solver again. Therefore its contents
is linked statically to your executable "pro" and the library is not
needed to resolve any symbols. Hence the linker places no reference in
"pro".

Best
Klaus

Am 21.10.2013 22:33, schrieb Anthony Vergottis:
> When running ./pro the program works as expected, but when the .so
> file is recompiled ./pro does not show any change. Change occurs
> only when the makefile is rerun. When moving ${LIBS} before
> ${PETSC_LIB} there is no change as well.
> 
> 
> On 21 October 2013 21:19, Barry Smith <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
> 
> 
> Please always make sure to include the mailing list in your list of
> emails. Otherwise it just goes to me, and I don't know what I am 
> talking about.
> 
> 
> Barry
> 
> On Oct 21, 2013, at 3:18 PM, Barry Smith <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
> 
>> 
>> What happens when you run
>> 
>> ./pro
>> 
>> ?
>> 
>> Try putting ${LIBS}  before the ${PETSC_LIB}  in the makefile
>> 
>> Barry
>> 
>> 
>> On Oct 21, 2013, at 3:10 PM, Anthony Vergottis
> <a.vergottis at ucl.ac.uk <mailto:a.vergottis at ucl.ac.uk>> wrote:
>> 
>>> OK this is the makefile,
>>> 
>>> include ${PETSC_DIR}/conf/variables include
>>> ${PETSC_DIR}/conf/rules 
>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>>> 
>>> 
>>> pro: pro.o 2D_CBS_Solver.o ${CLINKER} -o pro pro.o
>>> 2D_CBS_Solver.o ${PETSC_LIB}
> ${LIBS}
>>> ${RM} pro.o 2D_CBS_Solver.o
>>> 
>>> 
>>> The terminal output (in bold are my commands),
>>> 
>>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ g++ -shared -fPIC -o
> libpro.so 2D_CBS_Solver.cpp
>>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ls -l total 84 
>>> -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:56
>>> 2D_CBS_Solver.cpp -rw-rw-r-- 1 adonis adonis   144 Oct 21 20:50
>>> 2D_CBS_Solver.cpp~ -rw-rw-r-- 1 adonis adonis   429 Oct 21
>>> 19:13 2D_CBS_Solver.h drwxrwxr-x 2 adonis adonis  4096 Oct 18
>>> 14:31 include -rwxrwxr-x 1 adonis adonis  7968 Oct 21 21:02
>>> libpro.so -rw------- 1 adonis adonis   291 Oct 21 21:01
>>> Makefile -rw------- 1 adonis adonis   291 Oct 21 21:01
>>> Makefile~ -rwxrwxr-x 1 adonis adonis 42896 Oct 21 21:02 pro 
>>> -rw-rw-r-- 1 adonis adonis   940 Oct 21 20:42 pro.cpp 
>>> drwxrwxr-x 2 adonis adonis  4096 Oct 21 20:54 Solvers 
>>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ file libpro.so 
>>> libpro.so: ELF 64-bit LSB shared object, x86-64, version 1
> (SYSV), dynamically linked, 
> BuildID[sha1]=0x928d227f5816b3de3329f4364a688b6146df8c60, not
> stripped
>>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ make pro 
>>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx
>
>>> 
- -o pro.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -g  -fPIC 
> -I/home/adonis/Documents/petsc-3.4.2/include 
> -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include 
> -D__INSDIR__= pro.cpp
>>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx
>
>>> 
- -o 2D_CBS_Solver.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
> -Wno-unknown-pragmas -g  -fPIC 
> -I/home/adonis/Documents/petsc-3.4.2/include 
> -I/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/include 
> -D__INSDIR__= 2D_CBS_Solver.cpp
>>> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/bin/mpicxx
>
>>> 
- -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g
> -o pro pro.o 2D_CBS_Solver.o 
> -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib
>
> 
- -L/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib  -lpetsc
> -Wl,-rpath,/home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib
>
> 
- -lflapack -lfblas -lX11 -lpthread -lparmetis -lmetis -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.6 
> -L/usr/lib/gcc/x86_64-linux-gnu/4.6 
> -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu 
> -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu
> -lmpichf90 -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpichcxx
> -lstdc++ -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl 
> -L/home/adonis/Dropbox/ParallelSolver 
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>>> /bin/rm -f pro.o 2D_CBS_Solver.o 
>>> adonis at LinuxBox:~/Dropbox/ParallelSolver$ ldd pro 
>>> linux-vdso.so.1 =>  (0x00007fff1dffe000) libpetsc.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libpetsc.so
>
> 
(0x00007f24f4da6000)
>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f24f4b88000)
>>> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (0x00007f24f4888000)
>>> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x00007f24f4672000)
>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
>>> (0x00007f24f42b2000) libX11.so.6 =>
>>> /usr/lib/x86_64-linux-gnu/libX11.so.6
> (0x00007f24f3f7d000)
>>> libparmetis.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libparmetis.so
>
> 
(0x00007f24f3d2f000)
>>> libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3
> (0x00007f24f3a17000)
>>> libmpich.so.10 =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpich.so.10
>
> 
(0x00007f24f364b000)
>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> (0x00007f24f3447000)
>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
>>> (0x00007f24f314a000) /lib64/ld-linux-x86-64.so.2
>>> (0x00007f24f6176000) libxcb.so.1 =>
>>> /usr/lib/x86_64-linux-gnu/libxcb.so.1
> (0x00007f24f2f2c000)
>>> libmetis.so =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmetis.so
>
> 
(0x00007f24f2cb0000)
>>> libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
> (0x00007f24f2a7a000)
>>> libmpl.so.1 =>
> /home/adonis/Documents/petsc-3.4.2/linux-gnu-c-debug/lib/libmpl.so.1
>
> 
(0x00007f24f2875000)
>>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
> (0x00007f24f266c000)
>>> libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6
> (0x00007f24f2469000)
>>> libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6
> (0x00007f24f2263000)
>>> 
>>> 
>>> I hope this helps and is more clear. The  libpro.so object for
> some reason is not being linked.
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 21 October 2013 20:56, Barry Smith <bsmith at mcs.anl.gov
> <mailto:bsmith at mcs.anl.gov>> wrote:
>>> 
>>> Please cut and paste everything you've done in this process,
>>> the
> make, the running of ldd etc
>>> 
>>> Barry
>>> 
>>> On Oct 21, 2013, at 2:53 PM, Anthony Vergottis
> <a.vergottis at ucl.ac.uk <mailto:a.vergottis at ucl.ac.uk>> wrote:
>>> 
>>>> Thanks for the replies. Unfortunatly this has still not
>>>> worked.
> When using the ldd command the shared library libpro.so is not 
> listed there. Any other ideas why this may be not linking?
>>>> 
>>>> Thanks again.
>>>> 
>>>> 
>>>> On 21 October 2013 20:35, Satish Balay <balay at mcs.anl.gov
> <mailto:balay at mcs.anl.gov>> wrote:
>>>> On Mon, 21 Oct 2013, Anthony Vergottis wrote:
>>>> 
>>>>> Dear All,
>>>>> 
>>>>> I was wondering how does one include a .so object when
> compiling with a
>>>>> PETSc makefile? I have made various .so object for many
> projects and they
>>>>> work properly by specifying the -L location and the
>>>>> -lLibrary
> name. But
>>>>> when I try to link the .so object via a PETSc make file
>>>>> (after
> compilation
>>>>> with no errors) the ldd command does not show the library
>>>>> as
> being linked.
>>>>> 
>>>>> Is there any internal behaviour of PETSc I should be aware
>>>>> of?
> Have tried
>>>>> everything possible.
>>>>> 
>>>>> I have configure PETSc with the following flags (if this
>>>>> helps).
>>>>> 
>>>>> --download-mpich --with-clanguage=c++
>>>>> --download-f-blas-lapack --download-metis
>>>>> --download-parmetis --with-shared-libraries=1 
>>>>> --with-dynamic-loading=1
>>>>> 
>>>>> This is the sample makefile:
>>>>> 
>>>>> include ${PETSC_DIR}/conf/variables include
>>>>> ${PETSC_DIR}/conf/rules 
>>>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver -lpro
>>>> 
>>>> you need:
>>>> 
>>>> LIBS=-L/home/adonis/Dropbox/ParallelSolver
> -Wl,-rpath,/home/adonis/Dropbox/ParallelSolver -lpro
>>>> 
>>>> Satish
>>>> 
>>>>> 
>>>>> pro: pro.o 2D_CBS_Solver.o ${CLINKER} -o pro pro.o
>>>>> 2D_CBS_Solver.o ${PETSC_LIB}
> ${LIBS}
>>>>> ${RM} pro.o 2D_CBS_Solver.o
>>>>> 
>>>>> After this makefile is run still ldd does not show
>>>>> linkage.
>>>>> 
>>>>> Any help would be appreciated. Thanks in advance.
>>>>> 
>>>>> Best regards, Anthony
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- Anthony Vergottis
>>>> 
>>>> PhD Student Department of Mechanical Engineering University
>>>> College London Gower Street London WC1E 6BT
>>>> 
>>>> E-mail: a.vergottis at ucl.ac.uk <mailto:a.vergottis at ucl.ac.uk> 
>>>> Tel (UK Mobile): (+44) 079 1263 0064
> <tel:%28%2B44%29%20079%201263%200064>
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- Anthony Vergottis
>>> 
>>> PhD Student Department of Mechanical Engineering University
>>> College London Gower Street London WC1E 6BT
>>> 
>>> E-mail: a.vergottis at ucl.ac.uk <mailto:a.vergottis at ucl.ac.uk> 
>>> Tel (UK Mobile): (+44) 079 1263 0064
> <tel:%28%2B44%29%20079%201263%200064>
>> 
> 
> 
> 
> 
> 
> 
> -- Anthony Vergottis
> 
> PhD Student Department of Mechanical Engineering University College
> London Gower Street London WC1E 6BT
> 
> E-mail: a.vergottis at ucl.ac.uk <mailto:i.stojanovic at ucl.ac.uk> Tel
> (UK Mobile): (+44) 079 1263 0064

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSZZFuAAoJEHGtO1Agqpgxi3oIALN83i7x9GR94zMqt5V8oBwx
9IA3vG7lRRHcS2EQrPIjQOiG/hPM0tvRUf8ZAE/wZjzgBi97Zm6OL+US16qDK56k
J10C1pFQvisULaJ6cfJLrkq8BxR/p/rhzggKwpI8bRyxvKgUoClSKqbZK4p0DxNQ
orSN7+Dokv1W4DFhyOBYqeRL/nmeXm7hC4Y/MF1fS2PGMVOJkE3ZdRO/3IK7FGUt
3moal1SOpdt9EranV/am9xvb0AbQ2+LjHdiQtBUGZlQwKWYSOiz3wFFdwsLoh9AP
BUvMyTHQ66YpN84DLuwcInfoVAZEwI75ywT34FsRHXkQ3pcoo+uRZjjhA/914HA=
=kYAt
-----END PGP SIGNATURE-----

From knepley at gmail.com  Mon Oct 21 15:46:09 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 21 Oct 2013 15:46:09 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
Message-ID: <CAMYG4Gm4HAT2e3srktiZHkGUfuB2oG3sJ3d=rO+ChwByAJnVwg@mail.gmail.com>

On Mon, Oct 21, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Matt,
>
>      I think you are running on 1 process where the DMDA doesn't have an
> optimized path, when I run on 2 processes the numbers indicate nothing
> proportional to dof* number of local points
>

Yes, I figured if it was not doing the right thing on 1, why go to more? :)

   Matt


> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 182480 VecScatterCreate_PtoS()
>
> dof = 8
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 176080 VecScatterCreate_PtoS()
>
> dof = 4
>
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 169680 VecScatterCreate_PtoS()
>
> dof = 2
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 166480 VecScatterCreate_PtoS()
>
> dof =2 grid is 50 by 50 instead of 100 by 100
>
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter
> [0] 7 6352 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 43952 VecScatterCreate_PtoS()
>
> The IS creation in the DMDA is far more troubling
>
> /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
>
> dof = 2
>
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 81600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 42016 ISLocalToGlobalMappingCreate()
>
> dof = 4
>
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 163200 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 82816 ISLocalToGlobalMappingCreate()
>
> dof = 8
>
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 326400 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 164416 ISLocalToGlobalMappingCreate()
>
> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 489600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 246016 ISLocalToGlobalMappingCreate()
>
> Here the accessing of indices is at the point level (as well as block) and
> hence memory usage is proportional to dof* local number of grid points. Of
> course it is still only proportional to the vector size. There is some
> improvement we could make it; with a lot of refactoring we can remove the
> dof* completely, with a little refactoring we can bring it down to a single
> dof*local number of grid points.
>
>    I cannot understand why you are seeing memory usage 7 times more than a
> vector. That seems like a lot.
>
>    Barry
>
>
>
> On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> >
> >   The PETSc DMDA object greedily allocates several arrays of data used
> to set up the communication and other things like local to global mappings
> even before you create any vectors. This is why you see this big bump in
> memory usage.
> >
> >   BUT I don't think it should be any worse in 3.4 than in 3.3 or
> earlier; at least we did not intend to make it worse. Are you sure it is
> using more memory than in 3.3
> >
> >   In order for use to decrease the memory usage of the DMDA setup it
> would be helpful if we knew which objects created within it used the most
> memory.  There is some sloppiness in that routine of not reusing memory as
> well as could be, not sure how much difference that would make.
> >
> >
> >   Barry
> >
> >
> >
> > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> >
> >> Dear list members,
> >>
> >> I have noticed strange memory consumption after upgrading to 3.4
> series. I
> >> never had time to properly investigate, but here is what happens [yes,
> this
> >> might be a petsc4py issue, but I doubt it] is
> >>
> >> # helpers contains _ProcessMemoryInfoProc routine which just digs the
> memory
> >> # usage data from /proc
> >> import helpers
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> from petsc4py import PETSc
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> da = PETSc.DA().create(sizes=[100,100,100],
> >>
>  proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >>                      boundary_type=[3,0,0],
> >>                      stencil_type=PETSc.DA.StencilType.BOX,
> >>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> vec=da.createGlobalVec()
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >>
> >> outputs
> >>
> >> 48 MiB / 49348 kB
> >> 48 MiB / 49360 kB
> >> 381 MiB / 446228 kB
> >> 435 MiB / 446228 kB
> >>
> >> Which is odd: size of the actual data to be stored in the da is just
> about 56
> >> megabytes, so why does creating the da consume 7 times that? And why
> does the
> >> DA reserve the memory in the first place? I thought memory only gets
> allocated
> >> once an associated vector is created and it indeed looks like the
> >> createGlobalVec call does indeed allocate the right amount of data. But
> what
> >> is that 330 MiB that DA().create() consumes? [It's actually the .setUp()
> >> method that does the consuming, but that's not of much use as it needs
> to be
> >> called before a vector can be created.]
> >>
> >> Cheers,
> >> Juha
> >>
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131021/4a5922ce/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct 21 17:12:32 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 21 Oct 2013 17:12:32 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <CAMYG4Gm4HAT2e3srktiZHkGUfuB2oG3sJ3d=rO+ChwByAJnVwg@mail.gmail.com>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
	<CAMYG4Gm4HAT2e3srktiZHkGUfuB2oG3sJ3d=rO+ChwByAJnVwg@mail.gmail.com>
Message-ID: <DF42955E-DA0B-42CB-A0E6-FC5C2C8A4EBE@mcs.anl.gov>


   Lots of optimizations in VecScatter where it matters (i.e. in parallel) but not sequential :-).   We actually should fix this :-(

   Barry

On Oct 21, 2013, at 3:46 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 21, 2013 at 3:23 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    Matt,
> 
>      I think you are running on 1 process where the DMDA doesn't have an optimized path, when I run on 2 processes the numbers indicate nothing proportional to dof* number of local points
> 
> Yes, I figured if it was not doing the right thing on 1, why go to more? :)
> 
>    Matt
>  
> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 182480 VecScatterCreate_PtoS()
> 
> dof = 8
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 176080 VecScatterCreate_PtoS()
> 
> dof = 4
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 169680 VecScatterCreate_PtoS()
> 
> dof = 2
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
> [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 166480 VecScatterCreate_PtoS()
> 
> dof =2 grid is 50 by 50 instead of 100 by 100
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
> [0] 7 6352 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 43952 VecScatterCreate_PtoS()
> 
> The IS creation in the DMDA is far more troubling
> 
> /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> 
> dof = 2
> 
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 81600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 42016 ISLocalToGlobalMappingCreate()
> 
> dof = 4
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 163200 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 82816 ISLocalToGlobalMappingCreate()
> 
> dof = 8
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 326400 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 164416 ISLocalToGlobalMappingCreate()
> 
> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 489600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 246016 ISLocalToGlobalMappingCreate()
> 
> Here the accessing of indices is at the point level (as well as block) and hence memory usage is proportional to dof* local number of grid points. Of course it is still only proportional to the vector size. There is some improvement we could make it; with a lot of refactoring we can remove the dof* completely, with a little refactoring we can bring it down to a single dof*local number of grid points.
> 
>    I cannot understand why you are seeing memory usage 7 times more than a vector. That seems like a lot.
> 
>    Barry
> 
> 
> 
> On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> >
> >   The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.
> >
> >   BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3
> >
> >   In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.
> >
> >
> >   Barry
> >
> >
> >
> > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> >
> >> Dear list members,
> >>
> >> I have noticed strange memory consumption after upgrading to 3.4 series. I
> >> never had time to properly investigate, but here is what happens [yes, this
> >> might be a petsc4py issue, but I doubt it] is
> >>
> >> # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
> >> # usage data from /proc
> >> import helpers
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> from petsc4py import PETSc
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> da = PETSc.DA().create(sizes=[100,100,100],
> >>                      proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >>                      boundary_type=[3,0,0],
> >>                      stencil_type=PETSc.DA.StencilType.BOX,
> >>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> vec=da.createGlobalVec()
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >>
> >> outputs
> >>
> >> 48 MiB / 49348 kB
> >> 48 MiB / 49360 kB
> >> 381 MiB / 446228 kB
> >> 435 MiB / 446228 kB
> >>
> >> Which is odd: size of the actual data to be stored in the da is just about 56
> >> megabytes, so why does creating the da consume 7 times that? And why does the
> >> DA reserve the memory in the first place? I thought memory only gets allocated
> >> once an associated vector is created and it indeed looks like the
> >> createGlobalVec call does indeed allocate the right amount of data. But what
> >> is that 330 MiB that DA().create() consumes? [It's actually the .setUp()
> >> method that does the consuming, but that's not of much use as it needs to be
> >> called before a vector can be created.]
> >>
> >> Cheers,
> >> Juha
> >>
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From gdiso at ustc.edu  Tue Oct 22 02:18:44 2013
From: gdiso at ustc.edu (Gong Ding)
Date: Tue, 22 Oct 2013 15:18:44 +0800 (CST)
Subject: [petsc-users] SNES line search basic
Message-ID: <26705281.308841382426324160.JavaMail.coremail@mail.ustc.edu>

Hi,
I updated petsc from 3.2 to 3.4. However, I found that the nonlinear solver previously worked well now failed with massage DIVERGED_LINE_SEARCH. 
My problem is extremely nonlinear, cubic line search may fail but Newton method works.
  
With petsc 3.2, I only use basic Newton by calling 
SNESLineSearchSet(snes, SNESLineSearchNo, PETSC_NULL) 
With 3.4 I call basic Newton by  
SNESLineSearchSetType(snes, SNESLINESEARCHBASIC).

However, petsc 3.4 still reports 
[0] SNESSolve_NEWTONLS(): fnorm=3.7500000000000006e-02, gnorm=3.7500000000000006e-02, ynorm=1.0000000000000000e+00, lssucceed=0
[0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.414213562373e+00 near zero implies found a local minimum  


It seems petsc 3.4 can not disable line search. Does anyone have the same problem?

Gong Ding
 



From juhaj at iki.fi  Tue Oct 22 03:57:15 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Tue, 22 Oct 2013 10:57:15 +0200
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
Message-ID: <1677196.9ZmfKVC8up@dhcp071>

Barry,

I seem to have touched a topic which goes way past my knowledge of PETSc 
internals, but it's very nice to see a thorough response nevertheless. Thank 
you. And Matthew, too.

After reading your suspicions about number of ranks, I tried with 1, 2 and 4 
and the memory use indeed seems to go down from 1:

juhaj at dhcp071> CMD='import helpers; procdata=helpers._ProcessMemoryInfoProc(); 
print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; from petsc4py 
import PETSc; procdata=helpers._ProcessMemoryInfoProc(); print 
procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; da = 
PETSc.DA().create(sizes=[100,100,100], 
proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE], boundary_type=[3,0,0], 
stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1, 
comm=PETSc.COMM_WORLD); procdata=helpers._ProcessMemoryInfoProc(); print 
procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]'
juhaj at dhcp071> mpirun -np 1 python -c "$CMD"
21 MiB / 22280 kB
21 MiB / 22304 kB
354 MiB / 419176 kB
juhaj at dhcp071> mpirun -np 2 python -c "$CMD"
22 MiB / 23276 kB
22 MiB / 23020 kB
22 MiB / 23300 kB
22 MiB / 23044 kB
141 MiB / 145324 kB
141 MiB / 145068 kB
juhaj at dhcp071> mpirun -np 4 python -c "$CMD"
22 MiB / 23292 kB
22 MiB / 23036 kB
22 MiB / 23316 kB
22 MiB / 23060 kB
22 MiB / 23316 kB
22 MiB / 23340 kB
22 MiB / 23044 kB
22 MiB / 23068 kB
81 MiB / 83716 kB
82 MiB / 83976 kB
81 MiB / 83964 kB
81 MiB / 83724 kB

As one would expect, 4 ranks needs more memory than 2 ranks, but quite 
unexpectedly, 1 rank needs more than 2! I guess you are right: the 1-rank-case 
is not optimised and quite frankly, I don't mind: I only ever run small tests 
with one rank. Unfortunately, trying to create the simplest possible scenario 
to illustrate my point, I used a small DA and just one rank, precisely to 
avoid the case where the excess memory would be due to MPI buffers or such. 
Looks like my plan backfired. ;)

But even still, my 53 MiB lattice, without any vectors created, takes 280 or 
320 MiB of memory ? down to <6 from the original 6.6.

I will test with 3.3 later today if I have the time, but I'm pretty sure 
things were "better" there.

On Monday 21 October 2013 15:23:01 Barry Smith wrote:
>    Matt,
> 
>      I think you are running on 1 process where the DMDA doesn't have an
> optimized path, when I run on 2 processes the numbers indicate nothing
> proportional to dof* number of local points
> 
> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 182480 VecScatterCreate_PtoS()
> 
> dof = 8
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 176080 VecScatterCreate_PtoS()
> 
> dof = 4
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 169680 VecScatterCreate_PtoS()
> 
> dof = 2
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter [0] 7 21344 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 166480 VecScatterCreate_PtoS()
> 
> dof =2 grid is 50 by 50 instead of 100 by 100
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> VecScatter [0] 7 6352 VecScatterCreate()
> [0] 2 32 VecScatterCreateCommon_PtoS()
> [0] 39 43952 VecScatterCreate_PtoS()
> 
> The IS creation in the DMDA is far more troubling
> 
> /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> 
> dof = 2
> 
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 81600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 42016 ISLocalToGlobalMappingCreate()
> 
> dof = 4
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 163200 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 82816 ISLocalToGlobalMappingCreate()
> 
> dof = 8
> 
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 326400 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 164416 ISLocalToGlobalMappingCreate()
> 
> dof = 12
> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> [0] 1 20400 ISBlockSetIndices_Block()
> [0] 15 3760 ISCreate()
> [0] 4 128 ISCreate_Block()
> [0] 1 16 ISCreate_Stride()
> [0] 2 489600 ISGetIndices_Block()
> [0] 1 20400 ISLocalToGlobalMappingBlock()
> [0] 7 246016 ISLocalToGlobalMappingCreate()
> 
> Here the accessing of indices is at the point level (as well as block) and
> hence memory usage is proportional to dof* local number of grid points. Of
> course it is still only proportional to the vector size. There is some
> improvement we could make it; with a lot of refactoring we can remove the
> dof* completely, with a little refactoring we can bring it down to a single
> dof*local number of grid points.
> 
>    I cannot understand why you are seeing memory usage 7 times more than a
> vector. That seems like a lot.
> 
>    Barry
> 
> On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >   The PETSc DMDA object greedily allocates several arrays of data used to
> >   set up the communication and other things like local to global mappings
> >   even before you create any vectors. This is why you see this big bump
> >   in memory usage.
> >   
> >   BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier;
> >   at least we did not intend to make it worse. Are you sure it is using
> >   more memory than in 3.3
> >   
> >   In order for use to decrease the memory usage of the DMDA setup it would
> >   be helpful if we knew which objects created within it used the most
> >   memory.  There is some sloppiness in that routine of not reusing memory
> >   as well as could be, not sure how much difference that would make.
> >   
> >   
> >   Barry
> > 
> > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> >> Dear list members,
> >> 
> >> I have noticed strange memory consumption after upgrading to 3.4 series.
> >> I
> >> never had time to properly investigate, but here is what happens [yes,
> >> this
> >> might be a petsc4py issue, but I doubt it] is
> >> 
> >> # helpers contains _ProcessMemoryInfoProc routine which just digs the
> >> memory # usage data from /proc
> >> import helpers
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> from petsc4py import PETSc
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> da = PETSc.DA().create(sizes=[100,100,100],
> >> 
> >>                      proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >>                      boundary_type=[3,0,0],
> >>                      stencil_type=PETSc.DA.StencilType.BOX,
> >>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> >> 
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> vec=da.createGlobalVec()
> >> procdata=helpers._ProcessMemoryInfoProc()
> >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> >> 
> >> outputs
> >> 
> >> 48 MiB / 49348 kB
> >> 48 MiB / 49360 kB
> >> 381 MiB / 446228 kB
> >> 435 MiB / 446228 kB
> >> 
> >> Which is odd: size of the actual data to be stored in the da is just
> >> about 56 megabytes, so why does creating the da consume 7 times that?
> >> And why does the DA reserve the memory in the first place? I thought
> >> memory only gets allocated once an associated vector is created and it
> >> indeed looks like the
> >> createGlobalVec call does indeed allocate the right amount of data. But
> >> what is that 330 MiB that DA().create() consumes? [It's actually the
> >> .setUp() method that does the consuming, but that's not of much use as
> >> it needs to be called before a vector can be created.]
> >> 
> >> Cheers,
> >> Juha

From christophe.ortiz at ciemat.es  Tue Oct 22 04:16:39 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 22 Oct 2013 11:16:39 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+7LrE7fg3p6kmoTELHL7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
Message-ID: <CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>

Hi Jed,

I added a third specie to my problem (step by step :-)).

Now my system is the following:

u_t - alpha u_xx +(k.u.v - q.w) = 0
v_t - beta v_xx +(k.u.v - q.w) = 0
w_t -(k.u.v - q.w) = 0


So I have 2 PDEs and 1 ODE. I put everything under IFunction and IJacobian.
For the PDEs I defined Dirichlet boundary conditions.

For this simple system, I have lot of problems, it does not converge.

I followed your suggestions and I removed any options from the code, and
put TSSetFromOptions(); at the end.
The default options are thus SNESNEWTONLS, KSPGMRES, LineSearch bt, ARKIMEX
3.

I obtain the following output:

      TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
 0 family='arkimex' scheme=0:'3' dt=4.190e-01
      TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
      TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
      TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01, 1
failures exceeds current TS allowed
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR:   !
[0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase
-ts_max_snes_failures or make negative to attempt recovery!



ThenI increase the max number of failures. When I do so, I get the
following error message:

      TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
 0 family='arkimex' scheme=0:'3' dt=4.190e-01
      TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
      TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
      TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01
retrying with dt=2.301e-01
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.301e-01
retrying with dt=5.753e-02
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.753e-02
retrying with dt=1.438e-02
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 1.438e-02
retrying with dt=3.596e-03
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 3.596e-03
retrying with dt=8.990e-04
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 8.990e-04
retrying with dt=2.247e-04
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.247e-04
retrying with dt=5.619e-05
      TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.619e-05
retrying with dt=1.405e-05

...
...
      TSAdapt 'basic': step 105 stage rejected t=3.06853    +5.679e-179
retrying with dt=1.420e-179
      TSAdapt 'basic': step 105 stage rejected t=3.06853    +1.420e-179
retrying with dt=3.550e-180
      TSAdapt 'basic': step 105 stage rejected t=3.06853    +3.550e-180
retrying with dt=8.874e-181
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 15 is not-a-number or infinite
at end of function: Parameter number 3!

It seems it is not able to recover, it decreases the timestep so much that
at the end you are left with something that it not a number, or 0.

Any idea how to solve that ?
Some time ago you told me it could be due to a problem associated with DAE
formulation for boundary conditions and methods with explicit stages:

*Is your problem similar to ex25?  (If I remember
correctly, that example uses a DAE formulation for boundary conditions
and thus has trouble with some methods with explicit stages.  There may
be more going on.)*
*
*
How can I reformulate my problem so I do not use DAE formulation for
boundary conditions ? For instance, if I want u=u* at i=0, or Neumann (or
Robin) boundary conditions ?
If due to method with explicit stages, which method do you recommend ? I
tried the FullyImplicit option for ARKIMEX, and it's even worse, it gets
slower.

Thanks a lot in advance for your help.
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/617177c6/attachment.html>

From knepley at gmail.com  Tue Oct 22 05:06:02 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 05:06:02 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <1677196.9ZmfKVC8up@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
	<1677196.9ZmfKVC8up@dhcp071>
Message-ID: <CAMYG4GnB+H+mOSWARhcWii=K7ELidPQPtQA+gp0W2Xp+129Skg@mail.gmail.com>

On Tue, Oct 22, 2013 at 3:57 AM, Juha J?ykk? <juhaj at iki.fi> wrote:

> Barry,
>
> I seem to have touched a topic which goes way past my knowledge of PETSc
> internals, but it's very nice to see a thorough response nevertheless.
> Thank
> you. And Matthew, too.
>
> After reading your suspicions about number of ranks, I tried with 1, 2 and
> 4
> and the memory use indeed seems to go down from 1:
>

I am now convinced that /proc is showing total memory ever allocated since
the OS is not
recovering any freed memory. If you want to see memory allocated, but not
freed, just
do not destroy the DA and run with -malloc_test.

   Matt


> juhaj at dhcp071> CMD='import helpers;
> procdata=helpers._ProcessMemoryInfoProc();
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; from
> petsc4py
> import PETSc; procdata=helpers._ProcessMemoryInfoProc(); print
> procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; da =
> PETSc.DA().create(sizes=[100,100,100],
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE], boundary_type=[3,0,0],
> stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1,
> comm=PETSc.COMM_WORLD); procdata=helpers._ProcessMemoryInfoProc(); print
> procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]'
> juhaj at dhcp071> mpirun -np 1 python -c "$CMD"
> 21 MiB / 22280 kB
> 21 MiB / 22304 kB
> 354 MiB / 419176 kB
> juhaj at dhcp071> mpirun -np 2 python -c "$CMD"
> 22 MiB / 23276 kB
> 22 MiB / 23020 kB
> 22 MiB / 23300 kB
> 22 MiB / 23044 kB
> 141 MiB / 145324 kB
> 141 MiB / 145068 kB
> juhaj at dhcp071> mpirun -np 4 python -c "$CMD"
> 22 MiB / 23292 kB
> 22 MiB / 23036 kB
> 22 MiB / 23316 kB
> 22 MiB / 23060 kB
> 22 MiB / 23316 kB
> 22 MiB / 23340 kB
> 22 MiB / 23044 kB
> 22 MiB / 23068 kB
> 81 MiB / 83716 kB
> 82 MiB / 83976 kB
> 81 MiB / 83964 kB
> 81 MiB / 83724 kB
>
> As one would expect, 4 ranks needs more memory than 2 ranks, but quite
> unexpectedly, 1 rank needs more than 2! I guess you are right: the
> 1-rank-case
> is not optimised and quite frankly, I don't mind: I only ever run small
> tests
> with one rank. Unfortunately, trying to create the simplest possible
> scenario
> to illustrate my point, I used a small DA and just one rank, precisely to
> avoid the case where the excess memory would be due to MPI buffers or such.
> Looks like my plan backfired. ;)
>
> But even still, my 53 MiB lattice, without any vectors created, takes 280
> or
> 320 MiB of memory ? down to <6 from the original 6.6.
>
> I will test with 3.3 later today if I have the time, but I'm pretty sure
> things were "better" there.
>
> On Monday 21 October 2013 15:23:01 Barry Smith wrote:
> >    Matt,
> >
> >      I think you are running on 1 process where the DMDA doesn't have an
> > optimized path, when I run on 2 processes the numbers indicate nothing
> > proportional to dof* number of local points
> >
> > dof = 12
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > VecScatter [0] 7 21344 VecScatterCreate()
> > [0] 2 32 VecScatterCreateCommon_PtoS()
> > [0] 39 182480 VecScatterCreate_PtoS()
> >
> > dof = 8
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > VecScatter [0] 7 21344 VecScatterCreate()
> > [0] 2 32 VecScatterCreateCommon_PtoS()
> > [0] 39 176080 VecScatterCreate_PtoS()
> >
> > dof = 4
> >
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > VecScatter [0] 7 21344 VecScatterCreate()
> > [0] 2 32 VecScatterCreateCommon_PtoS()
> > [0] 39 169680 VecScatterCreate_PtoS()
> >
> > dof = 2
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > VecScatter [0] 7 21344 VecScatterCreate()
> > [0] 2 32 VecScatterCreateCommon_PtoS()
> > [0] 39 166480 VecScatterCreate_PtoS()
> >
> > dof =2 grid is 50 by 50 instead of 100 by 100
> >
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > VecScatter [0] 7 6352 VecScatterCreate()
> > [0] 2 32 VecScatterCreateCommon_PtoS()
> > [0] 39 43952 VecScatterCreate_PtoS()
> >
> > The IS creation in the DMDA is far more troubling
> >
> > /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> >
> > dof = 2
> >
> > [0] 1 20400 ISBlockSetIndices_Block()
> > [0] 15 3760 ISCreate()
> > [0] 4 128 ISCreate_Block()
> > [0] 1 16 ISCreate_Stride()
> > [0] 2 81600 ISGetIndices_Block()
> > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > [0] 7 42016 ISLocalToGlobalMappingCreate()
> >
> > dof = 4
> >
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > [0] 1 20400 ISBlockSetIndices_Block()
> > [0] 15 3760 ISCreate()
> > [0] 4 128 ISCreate_Block()
> > [0] 1 16 ISCreate_Stride()
> > [0] 2 163200 ISGetIndices_Block()
> > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > [0] 7 82816 ISLocalToGlobalMappingCreate()
> >
> > dof = 8
> >
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > [0] 1 20400 ISBlockSetIndices_Block()
> > [0] 15 3760 ISCreate()
> > [0] 4 128 ISCreate_Block()
> > [0] 1 16 ISCreate_Stride()
> > [0] 2 326400 ISGetIndices_Block()
> > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > [0] 7 164416 ISLocalToGlobalMappingCreate()
> >
> > dof = 12
> > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > [0] 1 20400 ISBlockSetIndices_Block()
> > [0] 15 3760 ISCreate()
> > [0] 4 128 ISCreate_Block()
> > [0] 1 16 ISCreate_Stride()
> > [0] 2 489600 ISGetIndices_Block()
> > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > [0] 7 246016 ISLocalToGlobalMappingCreate()
> >
> > Here the accessing of indices is at the point level (as well as block)
> and
> > hence memory usage is proportional to dof* local number of grid points.
> Of
> > course it is still only proportional to the vector size. There is some
> > improvement we could make it; with a lot of refactoring we can remove the
> > dof* completely, with a little refactoring we can bring it down to a
> single
> > dof*local number of grid points.
> >
> >    I cannot understand why you are seeing memory usage 7 times more than
> a
> > vector. That seems like a lot.
> >
> >    Barry
> >
> > On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >   The PETSc DMDA object greedily allocates several arrays of data used
> to
> > >   set up the communication and other things like local to global
> mappings
> > >   even before you create any vectors. This is why you see this big bump
> > >   in memory usage.
> > >
> > >   BUT I don't think it should be any worse in 3.4 than in 3.3 or
> earlier;
> > >   at least we did not intend to make it worse. Are you sure it is using
> > >   more memory than in 3.3
> > >
> > >   In order for use to decrease the memory usage of the DMDA setup it
> would
> > >   be helpful if we knew which objects created within it used the most
> > >   memory.  There is some sloppiness in that routine of not reusing
> memory
> > >   as well as could be, not sure how much difference that would make.
> > >
> > >
> > >   Barry
> > >
> > > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > >> Dear list members,
> > >>
> > >> I have noticed strange memory consumption after upgrading to 3.4
> series.
> > >> I
> > >> never had time to properly investigate, but here is what happens [yes,
> > >> this
> > >> might be a petsc4py issue, but I doubt it] is
> > >>
> > >> # helpers contains _ProcessMemoryInfoProc routine which just digs the
> > >> memory # usage data from /proc
> > >> import helpers
> > >> procdata=helpers._ProcessMemoryInfoProc()
> > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >> from petsc4py import PETSc
> > >> procdata=helpers._ProcessMemoryInfoProc()
> > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >> da = PETSc.DA().create(sizes=[100,100,100],
> > >>
> > >>
>  proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > >>                      boundary_type=[3,0,0],
> > >>                      stencil_type=PETSc.DA.StencilType.BOX,
> > >>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > >>
> > >> procdata=helpers._ProcessMemoryInfoProc()
> > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >> vec=da.createGlobalVec()
> > >> procdata=helpers._ProcessMemoryInfoProc()
> > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > >>
> > >> outputs
> > >>
> > >> 48 MiB / 49348 kB
> > >> 48 MiB / 49360 kB
> > >> 381 MiB / 446228 kB
> > >> 435 MiB / 446228 kB
> > >>
> > >> Which is odd: size of the actual data to be stored in the da is just
> > >> about 56 megabytes, so why does creating the da consume 7 times that?
> > >> And why does the DA reserve the memory in the first place? I thought
> > >> memory only gets allocated once an associated vector is created and it
> > >> indeed looks like the
> > >> createGlobalVec call does indeed allocate the right amount of data.
> But
> > >> what is that 330 MiB that DA().create() consumes? [It's actually the
> > >> .setUp() method that does the consuming, but that's not of much use as
> > >> it needs to be called before a vector can be created.]
> > >>
> > >> Cheers,
> > >> Juha
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/a9b9cc4b/attachment-0001.html>

From juhaj at iki.fi  Tue Oct 22 05:12:31 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Tue, 22 Oct 2013 12:12:31 +0200
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <CAMYG4GnB+H+mOSWARhcWii=K7ELidPQPtQA+gp0W2Xp+129Skg@mail.gmail.com>
References: <3178358.Nb05eOjZJN@dhcp071> <1677196.9ZmfKVC8up@dhcp071>
	<CAMYG4GnB+H+mOSWARhcWii=K7ELidPQPtQA+gp0W2Xp+129Skg@mail.gmail.com>
Message-ID: <2575406.bkxBsJKuUF@dhcp071>

On Tuesday 22 October 2013 05:06:02 Matthew Knepley wrote:
> On Tue, Oct 22, 2013 at 3:57 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > Barry,
> > 
> > I seem to have touched a topic which goes way past my knowledge of PETSc
> > internals, but it's very nice to see a thorough response nevertheless.
> > Thank
> > you. And Matthew, too.
> > 
> > After reading your suspicions about number of ranks, I tried with 1, 2 and
> > 4
> 
> > and the memory use indeed seems to go down from 1:
> I am now convinced that /proc is showing total memory ever allocated since
> the OS is not
> recovering any freed memory. If you want to see memory allocated, but not
> freed, just
> do not destroy the DA and run with -malloc_test.

I'm not sure what you mean here: I'm interested in the maximum amount of 
memory used at any point during the program execution. /proc is supposed to 
know that. And for a longer run, ps, top and /proc do indeed agree, so I think 
I have the right numbers.

Why the peak? Because I'm running on several machines where I get killed for 
exceeding a memory limit. Sometimes the limit is on VSZ, which is not a 
problem, but sometimes the limit is on RSS, which does present a problem, 
especially on some machines where there is no swap, so I need to stay below 
physical main memory limit all the time. It some of the memory gets freed 
later, it is of no use to me because by then I'm dead.

If I misunderstood something, please point it out. ;)

Cheers,
Juha

> 
>    Matt
> 
> > juhaj at dhcp071> CMD='import helpers;
> > procdata=helpers._ProcessMemoryInfoProc();
> > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; from
> > petsc4py
> > import PETSc; procdata=helpers._ProcessMemoryInfoProc(); print
> > procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; da =
> > PETSc.DA().create(sizes=[100,100,100],
> > proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > boundary_type=[3,0,0],
> > stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1,
> > comm=PETSc.COMM_WORLD); procdata=helpers._ProcessMemoryInfoProc(); print
> > procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]'
> > juhaj at dhcp071> mpirun -np 1 python -c "$CMD"
> > 21 MiB / 22280 kB
> > 21 MiB / 22304 kB
> > 354 MiB / 419176 kB
> > juhaj at dhcp071> mpirun -np 2 python -c "$CMD"
> > 22 MiB / 23276 kB
> > 22 MiB / 23020 kB
> > 22 MiB / 23300 kB
> > 22 MiB / 23044 kB
> > 141 MiB / 145324 kB
> > 141 MiB / 145068 kB
> > juhaj at dhcp071> mpirun -np 4 python -c "$CMD"
> > 22 MiB / 23292 kB
> > 22 MiB / 23036 kB
> > 22 MiB / 23316 kB
> > 22 MiB / 23060 kB
> > 22 MiB / 23316 kB
> > 22 MiB / 23340 kB
> > 22 MiB / 23044 kB
> > 22 MiB / 23068 kB
> > 81 MiB / 83716 kB
> > 82 MiB / 83976 kB
> > 81 MiB / 83964 kB
> > 81 MiB / 83724 kB
> > 
> > As one would expect, 4 ranks needs more memory than 2 ranks, but quite
> > unexpectedly, 1 rank needs more than 2! I guess you are right: the
> > 1-rank-case
> > is not optimised and quite frankly, I don't mind: I only ever run small
> > tests
> > with one rank. Unfortunately, trying to create the simplest possible
> > scenario
> > to illustrate my point, I used a small DA and just one rank, precisely to
> > avoid the case where the excess memory would be due to MPI buffers or
> > such.
> > Looks like my plan backfired. ;)
> > 
> > But even still, my 53 MiB lattice, without any vectors created, takes 280
> > or
> > 320 MiB of memory ? down to <6 from the original 6.6.
> > 
> > I will test with 3.3 later today if I have the time, but I'm pretty sure
> > things were "better" there.
> > 
> > On Monday 21 October 2013 15:23:01 Barry Smith wrote:
> > >    Matt,
> > >    
> > >      I think you are running on 1 process where the DMDA doesn't have an
> > > 
> > > optimized path, when I run on 2 processes the numbers indicate nothing
> > > proportional to dof* number of local points
> > > 
> > > dof = 12
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > VecScatter [0] 7 21344 VecScatterCreate()
> > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > [0] 39 182480 VecScatterCreate_PtoS()
> > > 
> > > dof = 8
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > VecScatter [0] 7 21344 VecScatterCreate()
> > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > [0] 39 176080 VecScatterCreate_PtoS()
> > > 
> > > dof = 4
> > > 
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > VecScatter [0] 7 21344 VecScatterCreate()
> > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > [0] 39 169680 VecScatterCreate_PtoS()
> > > 
> > > dof = 2
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > VecScatter [0] 7 21344 VecScatterCreate()
> > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > [0] 39 166480 VecScatterCreate_PtoS()
> > > 
> > > dof =2 grid is 50 by 50 instead of 100 by 100
> > > 
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > VecScatter [0] 7 6352 VecScatterCreate()
> > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > [0] 39 43952 VecScatterCreate_PtoS()
> > > 
> > > The IS creation in the DMDA is far more troubling
> > > 
> > > /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > > 
> > > dof = 2
> > > 
> > > [0] 1 20400 ISBlockSetIndices_Block()
> > > [0] 15 3760 ISCreate()
> > > [0] 4 128 ISCreate_Block()
> > > [0] 1 16 ISCreate_Stride()
> > > [0] 2 81600 ISGetIndices_Block()
> > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > [0] 7 42016 ISLocalToGlobalMappingCreate()
> > > 
> > > dof = 4
> > > 
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > > [0] 1 20400 ISBlockSetIndices_Block()
> > > [0] 15 3760 ISCreate()
> > > [0] 4 128 ISCreate_Block()
> > > [0] 1 16 ISCreate_Stride()
> > > [0] 2 163200 ISGetIndices_Block()
> > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > [0] 7 82816 ISLocalToGlobalMappingCreate()
> > > 
> > > dof = 8
> > > 
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > > [0] 1 20400 ISBlockSetIndices_Block()
> > > [0] 15 3760 ISCreate()
> > > [0] 4 128 ISCreate_Block()
> > > [0] 1 16 ISCreate_Stride()
> > > [0] 2 326400 ISGetIndices_Block()
> > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > [0] 7 164416 ISLocalToGlobalMappingCreate()
> > > 
> > > dof = 12
> > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
> > > [0] 1 20400 ISBlockSetIndices_Block()
> > > [0] 15 3760 ISCreate()
> > > [0] 4 128 ISCreate_Block()
> > > [0] 1 16 ISCreate_Stride()
> > > [0] 2 489600 ISGetIndices_Block()
> > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > [0] 7 246016 ISLocalToGlobalMappingCreate()
> > > 
> > > Here the accessing of indices is at the point level (as well as block)
> > 
> > and
> > 
> > > hence memory usage is proportional to dof* local number of grid points.
> > 
> > Of
> > 
> > > course it is still only proportional to the vector size. There is some
> > > improvement we could make it; with a lot of refactoring we can remove
> > > the
> > > dof* completely, with a little refactoring we can bring it down to a
> > 
> > single
> > 
> > > dof*local number of grid points.
> > > 
> > >    I cannot understand why you are seeing memory usage 7 times more than
> > 
> > a
> > 
> > > vector. That seems like a lot.
> > > 
> > >    Barry
> > > 
> > > On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >   The PETSc DMDA object greedily allocates several arrays of data used
> > 
> > to
> > 
> > > >   set up the communication and other things like local to global
> > 
> > mappings
> > 
> > > >   even before you create any vectors. This is why you see this big
> > > >   bump
> > > >   in memory usage.
> > > >   
> > > >   BUT I don't think it should be any worse in 3.4 than in 3.3 or
> > 
> > earlier;
> > 
> > > >   at least we did not intend to make it worse. Are you sure it is
> > > >   using
> > > >   more memory than in 3.3
> > > >   
> > > >   In order for use to decrease the memory usage of the DMDA setup it
> > 
> > would
> > 
> > > >   be helpful if we knew which objects created within it used the most
> > > >   memory.  There is some sloppiness in that routine of not reusing
> > 
> > memory
> > 
> > > >   as well as could be, not sure how much difference that would make.
> > > >   
> > > >   
> > > >   Barry
> > > > 
> > > > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > > >> Dear list members,
> > > >> 
> > > >> I have noticed strange memory consumption after upgrading to 3.4
> > 
> > series.
> > 
> > > >> I
> > > >> never had time to properly investigate, but here is what happens
> > > >> [yes,
> > > >> this
> > > >> might be a petsc4py issue, but I doubt it] is
> > > >> 
> > > >> # helpers contains _ProcessMemoryInfoProc routine which just digs the
> > > >> memory # usage data from /proc
> > > >> import helpers
> > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > >> from petsc4py import PETSc
> > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > >> da = PETSc.DA().create(sizes=[100,100,100],
> >  
> >  proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> >  
> > > >>                      boundary_type=[3,0,0],
> > > >>                      stencil_type=PETSc.DA.StencilType.BOX,
> > > >>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
> > > >> 
> > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > >> vec=da.createGlobalVec()
> > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > >> 
> > > >> outputs
> > > >> 
> > > >> 48 MiB / 49348 kB
> > > >> 48 MiB / 49360 kB
> > > >> 381 MiB / 446228 kB
> > > >> 435 MiB / 446228 kB
> > > >> 
> > > >> Which is odd: size of the actual data to be stored in the da is just
> > > >> about 56 megabytes, so why does creating the da consume 7 times that?
> > > >> And why does the DA reserve the memory in the first place? I thought
> > > >> memory only gets allocated once an associated vector is created and
> > > >> it
> > > >> indeed looks like the
> > > >> createGlobalVec call does indeed allocate the right amount of data.
> > 
> > But
> > 
> > > >> what is that 330 MiB that DA().create() consumes? [It's actually the
> > > >> .setUp() method that does the consuming, but that's not of much use
> > > >> as
> > > >> it needs to be called before a vector can be created.]
> > > >> 
> > > >> Cheers,
> > > >> Juha

From knepley at gmail.com  Tue Oct 22 05:47:04 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 05:47:04 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <2575406.bkxBsJKuUF@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071> <1677196.9ZmfKVC8up@dhcp071>
	<CAMYG4GnB+H+mOSWARhcWii=K7ELidPQPtQA+gp0W2Xp+129Skg@mail.gmail.com>
	<2575406.bkxBsJKuUF@dhcp071>
Message-ID: <CAMYG4GnYwzKRWrt0XnkcD3a+zKXZCC+NJkpdK4sQTVZXUQuKHw@mail.gmail.com>

On Tue, Oct 22, 2013 at 5:12 AM, Juha J?ykk? <juhaj at iki.fi> wrote:

> On Tuesday 22 October 2013 05:06:02 Matthew Knepley wrote:
> > On Tue, Oct 22, 2013 at 3:57 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > > Barry,
> > >
> > > I seem to have touched a topic which goes way past my knowledge of
> PETSc
> > > internals, but it's very nice to see a thorough response nevertheless.
> > > Thank
> > > you. And Matthew, too.
> > >
> > > After reading your suspicions about number of ranks, I tried with 1, 2
> and
> > > 4
> >
> > > and the memory use indeed seems to go down from 1:
> > I am now convinced that /proc is showing total memory ever allocated
> since
> > the OS is not
> > recovering any freed memory. If you want to see memory allocated, but not
> > freed, just
> > do not destroy the DA and run with -malloc_test.
>
> I'm not sure what you mean here: I'm interested in the maximum amount of
> memory used at any point during the program execution. /proc is supposed to
> know that. And for a longer run, ps, top and /proc do indeed agree, so I
> think
> I have the right numbers.
>
> Why the peak? Because I'm running on several machines where I get killed
> for
> exceeding a memory limit. Sometimes the limit is on VSZ, which is not a
> problem, but sometimes the limit is on RSS, which does present a problem,
> especially on some machines where there is no swap, so I need to stay below
> physical main memory limit all the time. It some of the memory gets freed
> later, it is of no use to me because by then I'm dead.
>
> If I misunderstood something, please point it out. ;)
>

We sometimes allocate temporary memory that we free afterwards (like 2
vectors which
we use to setup the scatters). It will not be allocated at the same time as
other vectors,
but if the OS does not reclaim the memory, it will show up on RSS.

  Matt


> Cheers,
> Juha
>
> >
> >    Matt
> >
> > > juhaj at dhcp071> CMD='import helpers;
> > > procdata=helpers._ProcessMemoryInfoProc();
> > > print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; from
> > > petsc4py
> > > import PETSc; procdata=helpers._ProcessMemoryInfoProc(); print
> > > procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; da =
> > > PETSc.DA().create(sizes=[100,100,100],
> > > proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > > boundary_type=[3,0,0],
> > > stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1,
> > > comm=PETSc.COMM_WORLD); procdata=helpers._ProcessMemoryInfoProc();
> print
> > > procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]'
> > > juhaj at dhcp071> mpirun -np 1 python -c "$CMD"
> > > 21 MiB / 22280 kB
> > > 21 MiB / 22304 kB
> > > 354 MiB / 419176 kB
> > > juhaj at dhcp071> mpirun -np 2 python -c "$CMD"
> > > 22 MiB / 23276 kB
> > > 22 MiB / 23020 kB
> > > 22 MiB / 23300 kB
> > > 22 MiB / 23044 kB
> > > 141 MiB / 145324 kB
> > > 141 MiB / 145068 kB
> > > juhaj at dhcp071> mpirun -np 4 python -c "$CMD"
> > > 22 MiB / 23292 kB
> > > 22 MiB / 23036 kB
> > > 22 MiB / 23316 kB
> > > 22 MiB / 23060 kB
> > > 22 MiB / 23316 kB
> > > 22 MiB / 23340 kB
> > > 22 MiB / 23044 kB
> > > 22 MiB / 23068 kB
> > > 81 MiB / 83716 kB
> > > 82 MiB / 83976 kB
> > > 81 MiB / 83964 kB
> > > 81 MiB / 83724 kB
> > >
> > > As one would expect, 4 ranks needs more memory than 2 ranks, but quite
> > > unexpectedly, 1 rank needs more than 2! I guess you are right: the
> > > 1-rank-case
> > > is not optimised and quite frankly, I don't mind: I only ever run small
> > > tests
> > > with one rank. Unfortunately, trying to create the simplest possible
> > > scenario
> > > to illustrate my point, I used a small DA and just one rank, precisely
> to
> > > avoid the case where the excess memory would be due to MPI buffers or
> > > such.
> > > Looks like my plan backfired. ;)
> > >
> > > But even still, my 53 MiB lattice, without any vectors created, takes
> 280
> > > or
> > > 320 MiB of memory ? down to <6 from the original 6.6.
> > >
> > > I will test with 3.3 later today if I have the time, but I'm pretty
> sure
> > > things were "better" there.
> > >
> > > On Monday 21 October 2013 15:23:01 Barry Smith wrote:
> > > >    Matt,
> > > >
> > > >      I think you are running on 1 process where the DMDA doesn't
> have an
> > > >
> > > > optimized path, when I run on 2 processes the numbers indicate
> nothing
> > > > proportional to dof* number of local points
> > > >
> > > > dof = 12
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > > VecScatter [0] 7 21344 VecScatterCreate()
> > > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > > [0] 39 182480 VecScatterCreate_PtoS()
> > > >
> > > > dof = 8
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > > VecScatter [0] 7 21344 VecScatterCreate()
> > > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > > [0] 39 176080 VecScatterCreate_PtoS()
> > > >
> > > > dof = 4
> > > >
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > > VecScatter [0] 7 21344 VecScatterCreate()
> > > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > > [0] 39 169680 VecScatterCreate_PtoS()
> > > >
> > > > dof = 2
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > > VecScatter [0] 7 21344 VecScatterCreate()
> > > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > > [0] 39 166480 VecScatterCreate_PtoS()
> > > >
> > > > dof =2 grid is 50 by 50 instead of 100 by 100
> > > >
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> > > > VecScatter [0] 7 6352 VecScatterCreate()
> > > > [0] 2 32 VecScatterCreateCommon_PtoS()
> > > > [0] 39 43952 VecScatterCreate_PtoS()
> > > >
> > > > The IS creation in the DMDA is far more troubling
> > > >
> > > > /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
> IS
> > > >
> > > > dof = 2
> > > >
> > > > [0] 1 20400 ISBlockSetIndices_Block()
> > > > [0] 15 3760 ISCreate()
> > > > [0] 4 128 ISCreate_Block()
> > > > [0] 1 16 ISCreate_Stride()
> > > > [0] 2 81600 ISGetIndices_Block()
> > > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > > [0] 7 42016 ISLocalToGlobalMappingCreate()
> > > >
> > > > dof = 4
> > > >
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log |
> grep IS
> > > > [0] 1 20400 ISBlockSetIndices_Block()
> > > > [0] 15 3760 ISCreate()
> > > > [0] 4 128 ISCreate_Block()
> > > > [0] 1 16 ISCreate_Stride()
> > > > [0] 2 163200 ISGetIndices_Block()
> > > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > > [0] 7 82816 ISLocalToGlobalMappingCreate()
> > > >
> > > > dof = 8
> > > >
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log |
> grep IS
> > > > [0] 1 20400 ISBlockSetIndices_Block()
> > > > [0] 15 3760 ISCreate()
> > > > [0] 4 128 ISCreate_Block()
> > > > [0] 1 16 ISCreate_Stride()
> > > > [0] 2 326400 ISGetIndices_Block()
> > > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > > [0] 7 164416 ISLocalToGlobalMappingCreate()
> > > >
> > > > dof = 12
> > > > ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log |
> grep IS
> > > > [0] 1 20400 ISBlockSetIndices_Block()
> > > > [0] 15 3760 ISCreate()
> > > > [0] 4 128 ISCreate_Block()
> > > > [0] 1 16 ISCreate_Stride()
> > > > [0] 2 489600 ISGetIndices_Block()
> > > > [0] 1 20400 ISLocalToGlobalMappingBlock()
> > > > [0] 7 246016 ISLocalToGlobalMappingCreate()
> > > >
> > > > Here the accessing of indices is at the point level (as well as
> block)
> > >
> > > and
> > >
> > > > hence memory usage is proportional to dof* local number of grid
> points.
> > >
> > > Of
> > >
> > > > course it is still only proportional to the vector size. There is
> some
> > > > improvement we could make it; with a lot of refactoring we can remove
> > > > the
> > > > dof* completely, with a little refactoring we can bring it down to a
> > >
> > > single
> > >
> > > > dof*local number of grid points.
> > > >
> > > >    I cannot understand why you are seeing memory usage 7 times more
> than
> > >
> > > a
> > >
> > > > vector. That seems like a lot.
> > > >
> > > >    Barry
> > > >
> > > > On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > > >   The PETSc DMDA object greedily allocates several arrays of data
> used
> > >
> > > to
> > >
> > > > >   set up the communication and other things like local to global
> > >
> > > mappings
> > >
> > > > >   even before you create any vectors. This is why you see this big
> > > > >   bump
> > > > >   in memory usage.
> > > > >
> > > > >   BUT I don't think it should be any worse in 3.4 than in 3.3 or
> > >
> > > earlier;
> > >
> > > > >   at least we did not intend to make it worse. Are you sure it is
> > > > >   using
> > > > >   more memory than in 3.3
> > > > >
> > > > >   In order for use to decrease the memory usage of the DMDA setup
> it
> > >
> > > would
> > >
> > > > >   be helpful if we knew which objects created within it used the
> most
> > > > >   memory.  There is some sloppiness in that routine of not reusing
> > >
> > > memory
> > >
> > > > >   as well as could be, not sure how much difference that would
> make.
> > > > >
> > > > >
> > > > >   Barry
> > > > >
> > > > > On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
> > > > >> Dear list members,
> > > > >>
> > > > >> I have noticed strange memory consumption after upgrading to 3.4
> > >
> > > series.
> > >
> > > > >> I
> > > > >> never had time to properly investigate, but here is what happens
> > > > >> [yes,
> > > > >> this
> > > > >> might be a petsc4py issue, but I doubt it] is
> > > > >>
> > > > >> # helpers contains _ProcessMemoryInfoProc routine which just digs
> the
> > > > >> memory # usage data from /proc
> > > > >> import helpers
> > > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > >> from petsc4py import PETSc
> > > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > >> da = PETSc.DA().create(sizes=[100,100,100],
> > >
> > >  proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
> > >
> > > > >>                      boundary_type=[3,0,0],
> > > > >>                      stencil_type=PETSc.DA.StencilType.BOX,
> > > > >>                      dof=7, stencil_width=1,
> comm=PETSc.COMM_WORLD)
> > > > >>
> > > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > >> vec=da.createGlobalVec()
> > > > >> procdata=helpers._ProcessMemoryInfoProc()
> > > > >> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
> > > > >>
> > > > >> outputs
> > > > >>
> > > > >> 48 MiB / 49348 kB
> > > > >> 48 MiB / 49360 kB
> > > > >> 381 MiB / 446228 kB
> > > > >> 435 MiB / 446228 kB
> > > > >>
> > > > >> Which is odd: size of the actual data to be stored in the da is
> just
> > > > >> about 56 megabytes, so why does creating the da consume 7 times
> that?
> > > > >> And why does the DA reserve the memory in the first place? I
> thought
> > > > >> memory only gets allocated once an associated vector is created
> and
> > > > >> it
> > > > >> indeed looks like the
> > > > >> createGlobalVec call does indeed allocate the right amount of
> data.
> > >
> > > But
> > >
> > > > >> what is that 330 MiB that DA().create() consumes? [It's actually
> the
> > > > >> .setUp() method that does the consuming, but that's not of much
> use
> > > > >> as
> > > > >> it needs to be called before a vector can be created.]
> > > > >>
> > > > >> Cheers,
> > > > >> Juha
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/c880e430/attachment-0001.html>

From knepley at gmail.com  Tue Oct 22 05:57:27 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 05:57:27 -0500
Subject: [petsc-users] SNES line search basic
In-Reply-To: <26705281.308841382426324160.JavaMail.coremail@mail.ustc.edu>
References: <26705281.308841382426324160.JavaMail.coremail@mail.ustc.edu>
Message-ID: <CAMYG4G=2ijzjYPPBoyp8G8WOmXyEPrALJwWTY1kZPdDfKVYQMQ@mail.gmail.com>

On Tue, Oct 22, 2013 at 2:18 AM, Gong Ding <gdiso at ustc.edu> wrote:

> Hi,
> I updated petsc from 3.2 to 3.4. However, I found that the nonlinear
> solver previously worked well now failed with massage DIVERGED_LINE_SEARCH.
> My problem is extremely nonlinear, cubic line search may fail but Newton
> method works.
>
> With petsc 3.2, I only use basic Newton by calling
> SNESLineSearchSet(snes, SNESLineSearchNo, PETSC_NULL)
> With 3.4 I call basic Newton by
> SNESLineSearchSetType(snes, SNESLINESEARCHBASIC).
>
> However, petsc 3.4 still reports
> [0] SNESSolve_NEWTONLS(): fnorm=3.7500000000000006e-02,
> gnorm=3.7500000000000006e-02, ynorm=1.0000000000000000e+00, lssucceed=0
> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.414213562373e+00
> near zero implies found a local minimum
>
>
> It seems petsc 3.4 can not disable line search. Does anyone have the same
> problem?


Send -snes_view

   Matt


>
> Gong Ding
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/7d5e8e14/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 22 06:58:53 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 22 Oct 2013 06:58:53 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<CANBrw+7LrE7fg3p6kmoTELHL
	7cyOO_WCS9Rdr4Ej3CH9h7kq+Q@mail.gmail.com> <87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
Message-ID: <B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>


   Add -snes_monitor -snes_converged_reason -ksp_monitor -ksp_converged_reason -pc_type lu (send the output)

   Does it now get stuck at the same time step or does it get past it?   The most common cause of this problem is an incorrect Jacobian http://www.mcs.anl.gov/petsc/documentation/faq.html#newton

   Barry


On Oct 22, 2013, at 4:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es> wrote:

> Hi Jed,
> 
> I added a third specie to my problem (step by step :-)).
> 
> Now my system is the following:
> 
> u_t - alpha u_xx +(k.u.v - q.w) = 0
> v_t - beta v_xx +(k.u.v - q.w) = 0
> w_t -(k.u.v - q.w) = 0
> 
> 
> So I have 2 PDEs and 1 ODE. I put everything under IFunction and IJacobian.
> For the PDEs I defined Dirichlet boundary conditions.
> 
> For this simple system, I have lot of problems, it does not converge.
> 
> I followed your suggestions and I removed any options from the code, and put TSSetFromOptions(); at the end.
> The default options are thus SNESNEWTONLS, KSPGMRES, LineSearch bt, ARKIMEX 3.
> 
> I obtain the following output:
> 
>       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=    0 family='arkimex' scheme=0:'3' dt=4.190e-01 
>       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01 wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01 
>       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01 wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01 
>       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01 wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01, 1 failures exceeds current TS allowed
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR:   !
> [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE, increase -ts_max_snes_failures or make negative to attempt recovery!
> 
> 
> 
> ThenI increase the max number of failures. When I do so, I get the following error message:
> 
>       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=    0 family='arkimex' scheme=0:'3' dt=4.190e-01 
>       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01 wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01 
>       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01 wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01 
>       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01 wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01 retrying with dt=2.301e-01 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.301e-01 retrying with dt=5.753e-02 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.753e-02 retrying with dt=1.438e-02 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 1.438e-02 retrying with dt=3.596e-03 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 3.596e-03 retrying with dt=8.990e-04 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 8.990e-04 retrying with dt=2.247e-04 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.247e-04 retrying with dt=5.619e-05 
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.619e-05 retrying with dt=1.405e-05 
>      
> ...
> ...
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    +5.679e-179 retrying with dt=1.420e-179
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    +1.420e-179 retrying with dt=3.550e-180
>       TSAdapt 'basic': step 105 stage rejected t=3.06853    +3.550e-180 retrying with dt=8.874e-181
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Vec entry at local location 15 is not-a-number or infinite at end of function: Parameter number 3!
> 
> It seems it is not able to recover, it decreases the timestep so much that at the end you are left with something that it not a number, or 0.
> 
> Any idea how to solve that ?
> Some time ago you told me it could be due to a problem associated with DAE formulation for boundary conditions and methods with explicit stages:
> 
> Is your problem similar to ex25?  (If I remember
> correctly, that example uses a DAE formulation for boundary conditions
> and thus has trouble with some methods with explicit stages.  There may
> be more going on.)
> 
> How can I reformulate my problem so I do not use DAE formulation for boundary conditions ? For instance, if I want u=u* at i=0, or Neumann (or Robin) boundary conditions ?
> If due to method with explicit stages, which method do you recommend ? I tried the FullyImplicit option for ARKIMEX, and it's even worse, it gets slower.
> 
> Thanks a lot in advance for your help.
> Christophe
> 


From jedbrown at mcs.anl.gov  Tue Oct 22 07:25:41 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 22 Oct 2013 07:25:41 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <CAMYG4GnYwzKRWrt0XnkcD3a+zKXZCC+NJkpdK4sQTVZXUQuKHw@mail.gmail.com>
References: <3178358.Nb05eOjZJN@dhcp071> <1677196.9ZmfKVC8up@dhcp071>
	<CAMYG4GnB+H+mOSWARhcWii=K7ELidPQPtQA+gp0W2Xp+129Skg@mail.gmail.com>
	<2575406.bkxBsJKuUF@dhcp071>
	<CAMYG4GnYwzKRWrt0XnkcD3a+zKXZCC+NJkpdK4sQTVZXUQuKHw@mail.gmail.com>
Message-ID: <87ob6hpk56.fsf@mcs.anl.gov>

Matthew Knepley <knepley at gmail.com> writes:
> We sometimes allocate temporary memory that we free afterwards (like 2
> vectors which we use to setup the scatters). It will not be allocated
> at the same time as other vectors, but if the OS does not reclaim the
> memory, it will show up on RSS.

That should not show up in the information he's gathering.  Only the
peak allocated at any one time, not the cumulative value of all calls to
malloc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/5ad7fa15/attachment.pgp>

From ricardo.marquez.88 at gmail.com  Tue Oct 22 07:39:16 2013
From: ricardo.marquez.88 at gmail.com (Ricardo Marquez)
Date: Tue, 22 Oct 2013 10:39:16 -0200
Subject: [petsc-users] Problem with SLEPC Eigenvalue Problem Solver
Message-ID: <CAMtjCgTQhGnodNLud3Tg4DvnCvjKi5BgfT4_mX=orH+DYTJk7A@mail.gmail.com>

Hi, I'm trying to solve a vibration generalized eigenvalue problem like
this:

|  K    C |  =  lambda* | M  0 |
| C^t   0 |                    | 0  0 |

I have try a lot of convinations eigensolvers, KSP and PC, but I can't make
that it works.
I use the ST shift-and-invert to get the firsts 5 eigenvalues.
If I use PC or ST_PC = LU or CHOLESKY, I get the zero pivot message error.
If I use PC and ST_PC = NONE, the 5 eigenvalues that find the solver are
exactly the same and equal to the shift value.
If I use PC = jacobi or bjacobi, the eigenvalues get a very big value.

Does anyone knows how this kind of problem solve?

thanks!
-- 
Ricardo M?rquez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/e1ef902b/attachment.html>

From jroman at dsic.upv.es  Tue Oct 22 07:56:59 2013
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 22 Oct 2013 14:56:59 +0200
Subject: [petsc-users] [petsc-dev] Problem with SLEPC Eigenvalue Problem
	Solver
In-Reply-To: <CAMtjCgTQhGnodNLud3Tg4DvnCvjKi5BgfT4_mX=orH+DYTJk7A@mail.gmail.com>
References: <CAMtjCgTQhGnodNLud3Tg4DvnCvjKi5BgfT4_mX=orH+DYTJk7A@mail.gmail.com>
Message-ID: <A1599409-B87A-41F5-B9ED-D669B2C67406@dsic.upv.es>


El 22/10/2013, a las 14:39, Ricardo Marquez escribi?:

> Hi, I'm trying to solve a vibration generalized eigenvalue problem like this:
> 
> |  K    C |  =  lambda* | M  0 |
> | C^t   0 |                    | 0  0 |
> 
> I have try a lot of convinations eigensolvers, KSP and PC, but I can't make that it works.
> I use the ST shift-and-invert to get the firsts 5 eigenvalues.
> If I use PC or ST_PC = LU or CHOLESKY, I get the zero pivot message error.
> If I use PC and ST_PC = NONE, the 5 eigenvalues that find the solver are exactly the same and equal to the shift value.
> If I use PC = jacobi or bjacobi, the eigenvalues get a very big value.  
> 
> Does anyone knows how this kind of problem solve?
> 
> thanks!
> -- 
> Ricardo M?rquez


If C has a relatively small number of columns, you may consider formulating the problem as a constrained eigenproblem

   K x = lambda M x,  with  C^t x = 0

Try representing C as a Vec array and passing them to EPSSetDeflationSpace() before EPSSolve().

Jose


From christophe.ortiz at ciemat.es  Tue Oct 22 08:29:43 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 22 Oct 2013 15:29:43 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
Message-ID: <CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>

Hi Barry,

Thanks for suggestions. Here is the output.
Before using the options you suggest I get:

 TSAdapt 'basic': step 182 accepted t=0.341561   + 3.416e-02 wlte=1.1e-09
family='arkimex' scheme=0:'3' dt=3.757e-02
      TSAdapt 'basic': step 183 accepted t=0.375717   + 3.757e-02
wlte=9.05e-14 family='arkimex' scheme=0:'3' dt=4.133e-02
      TSAdapt 'basic': step 184 stage rejected t=0.413289   + 4.133e-02
retrying with dt=1.033e-02
      TSAdapt 'basic': step 184 stage rejected t=0.413289   + 1.033e-02
retrying with dt=2.583e-03
...
      TSAdapt 'basic': step 184 stage rejected t=0.413289   +2.212e-198
retrying with dt=5.529e-199
      TSAdapt 'basic': step 184 stage rejected t=0.413289   +5.529e-199
retrying with dt=1.382e-199
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 12 is not-a-number or infinite
at end of function: Parameter number 3!


 I re-checked my Jacobian several times, and I found few errors. But now I
think it is ok.
Then, with the options you suggest, it stops immediatly. Not even one
timestep:

    0 SNES Function norm 5.929210530071e+26
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Detected zero pivot in LU factorization:
see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
[0]PETSC ERROR: Zero pivot row 17 value 1.35114e-16 tolerance 2.22045e-14!
...
[0]PETSC ERROR: MatPivotCheck_none() line 589 in
src/mat/impls/aij/seq//home/u5751/petsc-3.4.1/include/petsc-private/matimpl.h
[0]PETSC ERROR: MatPivotCheck() line 608 in
src/mat/impls/aij/seq//home/u5751/petsc-3.4.1/include/petsc-private/matimpl.h
[0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1626 in
src/mat/impls/aij/seq/inode.c
[0]PETSC ERROR: MatLUFactorNumeric() line 2889 in src/mat/interface/matrix.c
[0]PETSC ERROR: PCSetUp_LU() line 152 in src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: PCSetUp() line 890 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 220 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
[0]PETSC ERROR: main() line 455 in src/ts/examples/tutorials/diffusion.c

Does this give you a hint ?

Christophe


CIEMAT
Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
Unidad de Materiales
Edificio 2 - Planta 0 - Despacho 28m
Avenida Complutense 40,
28040 Madrid, Spain
Tel: +34 91496 2582
Fax: +34 91346 6442

--
Q
Por favor, piense en el medio ambiente antes de imprimir este mensaje.
Please consider the environment before printing this email.


On Tue, Oct 22, 2013 at 1:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Add -snes_monitor -snes_converged_reason -ksp_monitor
> -ksp_converged_reason -pc_type lu (send the output)
>
>    Does it now get stuck at the same time step or does it get past it?
> The most common cause of this problem is an incorrect Jacobian
> http://www.mcs.anl.gov/petsc/documentation/faq.html#newton
>
>    Barry
>
>
> On Oct 22, 2013, at 4:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
> wrote:
>
> > Hi Jed,
> >
> > I added a third specie to my problem (step by step :-)).
> >
> > Now my system is the following:
> >
> > u_t - alpha u_xx +(k.u.v - q.w) = 0
> > v_t - beta v_xx +(k.u.v - q.w) = 0
> > w_t -(k.u.v - q.w) = 0
> >
> >
> > So I have 2 PDEs and 1 ODE. I put everything under IFunction and
> IJacobian.
> > For the PDEs I defined Dirichlet boundary conditions.
> >
> > For this simple system, I have lot of problems, it does not converge.
> >
> > I followed your suggestions and I removed any options from the code, and
> put TSSetFromOptions(); at the end.
> > The default options are thus SNESNEWTONLS, KSPGMRES, LineSearch bt,
> ARKIMEX 3.
> >
> > I obtain the following output:
> >
> >       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
>    0 family='arkimex' scheme=0:'3' dt=4.190e-01
> >       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
> wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
> >       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
> wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
> >       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
> wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01,
> 1 failures exceeds current TS allowed
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR:   !
> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE,
> increase -ts_max_snes_failures or make negative to attempt recovery!
> >
> >
> >
> > ThenI increase the max number of failures. When I do so, I get the
> following error message:
> >
> >       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
>    0 family='arkimex' scheme=0:'3' dt=4.190e-01
> >       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
> wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
> >       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
> wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
> >       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
> wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01
> retrying with dt=2.301e-01
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.301e-01
> retrying with dt=5.753e-02
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.753e-02
> retrying with dt=1.438e-02
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 1.438e-02
> retrying with dt=3.596e-03
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 3.596e-03
> retrying with dt=8.990e-04
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 8.990e-04
> retrying with dt=2.247e-04
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.247e-04
> retrying with dt=5.619e-05
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.619e-05
> retrying with dt=1.405e-05
> >
> > ...
> > ...
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +5.679e-179
> retrying with dt=1.420e-179
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +1.420e-179
> retrying with dt=3.550e-180
> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +3.550e-180
> retrying with dt=8.874e-181
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Floating point exception!
> > [0]PETSC ERROR: Vec entry at local location 15 is not-a-number or
> infinite at end of function: Parameter number 3!
> >
> > It seems it is not able to recover, it decreases the timestep so much
> that at the end you are left with something that it not a number, or 0.
> >
> > Any idea how to solve that ?
> > Some time ago you told me it could be due to a problem associated with
> DAE formulation for boundary conditions and methods with explicit stages:
> >
> > Is your problem similar to ex25?  (If I remember
> > correctly, that example uses a DAE formulation for boundary conditions
> > and thus has trouble with some methods with explicit stages.  There may
> > be more going on.)
> >
> > How can I reformulate my problem so I do not use DAE formulation for
> boundary conditions ? For instance, if I want u=u* at i=0, or Neumann (or
> Robin) boundary conditions ?
> > If due to method with explicit stages, which method do you recommend ? I
> tried the FullyImplicit option for ARKIMEX, and it's even worse, it gets
> slower.
> >
> > Thanks a lot in advance for your help.
> > Christophe
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/02c6ba4b/attachment-0001.html>

From knepley at gmail.com  Tue Oct 22 08:37:51 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 08:37:51 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
Message-ID: <CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>

On Tue, Oct 22, 2013 at 8:29 AM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

> Hi Barry,
>
> Thanks for suggestions. Here is the output.
> Before using the options you suggest I get:
>
>  TSAdapt 'basic': step 182 accepted t=0.341561   + 3.416e-02 wlte=1.1e-09
> family='arkimex' scheme=0:'3' dt=3.757e-02
>       TSAdapt 'basic': step 183 accepted t=0.375717   + 3.757e-02
> wlte=9.05e-14 family='arkimex' scheme=0:'3' dt=4.133e-02
>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 4.133e-02
> retrying with dt=1.033e-02
>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 1.033e-02
> retrying with dt=2.583e-03
> ...
>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +2.212e-198
> retrying with dt=5.529e-199
>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +5.529e-199
> retrying with dt=1.382e-199
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Vec entry at local location 12 is not-a-number or infinite
> at end of function: Parameter number 3!
>
>
>  I re-checked my Jacobian several times, and I found few errors. But now I
> think it is ok.
> Then, with the options you suggest, it stops immediatly. Not even one
> timestep:
>

Something is very wrong here. The Jacobian should at least have something
on the diagonal from the time derivative,
but the Laplacian (u, v) and q (w) also have diagonals.

   Matt


>     0 SNES Function norm 5.929210530071e+26
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Detected zero pivot in LU factorization:
> see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
> [0]PETSC ERROR: Zero pivot row 17 value 1.35114e-16 tolerance 2.22045e-14!
> ...
> [0]PETSC ERROR: MatPivotCheck_none() line 589 in
> src/mat/impls/aij/seq//home/u5751/petsc-3.4.1/include/petsc-private/matimpl.h
> [0]PETSC ERROR: MatPivotCheck() line 608 in
> src/mat/impls/aij/seq//home/u5751/petsc-3.4.1/include/petsc-private/matimpl.h
> [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ_Inode() line 1626 in
> src/mat/impls/aij/seq/inode.c
> [0]PETSC ERROR: MatLUFactorNumeric() line 2889 in
> src/mat/interface/matrix.c
> [0]PETSC ERROR: PCSetUp_LU() line 152 in src/ksp/pc/impls/factor/lu/lu.c
> [0]PETSC ERROR: PCSetUp() line 890 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 220 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
> [0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
> [0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
> [0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
> [0]PETSC ERROR: main() line 455 in src/ts/examples/tutorials/diffusion.c
>
> Does this give you a hint ?
>
> Christophe
>
>
> CIEMAT
> Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
> Unidad de Materiales
> Edificio 2 - Planta 0 - Despacho 28m
> Avenida Complutense 40,
> 28040 Madrid, Spain
> Tel: +34 91496 2582
> Fax: +34 91346 6442
>
> --
> Q
> Por favor, piense en el medio ambiente antes de imprimir este mensaje.
> Please consider the environment before printing this email.
>
>
> On Tue, Oct 22, 2013 at 1:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>    Add -snes_monitor -snes_converged_reason -ksp_monitor
>> -ksp_converged_reason -pc_type lu (send the output)
>>
>>    Does it now get stuck at the same time step or does it get past it?
>> The most common cause of this problem is an incorrect Jacobian
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#newton
>>
>>    Barry
>>
>>
>> On Oct 22, 2013, at 4:16 AM, Christophe Ortiz <christophe.ortiz at ciemat.es>
>> wrote:
>>
>> > Hi Jed,
>> >
>> > I added a third specie to my problem (step by step :-)).
>> >
>> > Now my system is the following:
>> >
>> > u_t - alpha u_xx +(k.u.v - q.w) = 0
>> > v_t - beta v_xx +(k.u.v - q.w) = 0
>> > w_t -(k.u.v - q.w) = 0
>> >
>> >
>> > So I have 2 PDEs and 1 ODE. I put everything under IFunction and
>> IJacobian.
>> > For the PDEs I defined Dirichlet boundary conditions.
>> >
>> > For this simple system, I have lot of problems, it does not converge.
>> >
>> > I followed your suggestions and I removed any options from the code,
>> and put TSSetFromOptions(); at the end.
>> > The default options are thus SNESNEWTONLS, KSPGMRES, LineSearch bt,
>> ARKIMEX 3.
>> >
>> > I obtain the following output:
>> >
>> >       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
>>    0 family='arkimex' scheme=0:'3' dt=4.190e-01
>> >       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
>> wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
>> >       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
>> wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
>> >       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
>> wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +
>> 9.206e-01, 1 failures exceeds current TS allowed
>> > [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> > [0]PETSC ERROR:   !
>> > [0]PETSC ERROR: TSStep has failed due to DIVERGED_NONLINEAR_SOLVE,
>> increase -ts_max_snes_failures or make negative to attempt recovery!
>> >
>> >
>> >
>> > ThenI increase the max number of failures. When I do so, I get the
>> following error message:
>> >
>> >       TSAdapt 'basic': step 101 accepted t=1.07438    + 3.223e-01 wlte=
>>    0 family='arkimex' scheme=0:'3' dt=4.190e-01
>> >       TSAdapt 'basic': step 102 accepted t=1.39669    + 4.190e-01
>> wlte=5.21e-13 family='arkimex' scheme=0:'3' dt=5.447e-01
>> >       TSAdapt 'basic': step 103 accepted t=1.8157     + 5.447e-01
>> wlte=1.64e-13 family='arkimex' scheme=0:'3' dt=7.081e-01
>> >       TSAdapt 'basic': step 104 accepted t=2.36041    + 7.081e-01
>> wlte=5.62e-14 family='arkimex' scheme=0:'3' dt=9.206e-01
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 9.206e-01
>> retrying with dt=2.301e-01
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.301e-01
>> retrying with dt=5.753e-02
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.753e-02
>> retrying with dt=1.438e-02
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 1.438e-02
>> retrying with dt=3.596e-03
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 3.596e-03
>> retrying with dt=8.990e-04
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 8.990e-04
>> retrying with dt=2.247e-04
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 2.247e-04
>> retrying with dt=5.619e-05
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    + 5.619e-05
>> retrying with dt=1.405e-05
>> >
>> > ...
>> > ...
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +5.679e-179
>> retrying with dt=1.420e-179
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +1.420e-179
>> retrying with dt=3.550e-180
>> >       TSAdapt 'basic': step 105 stage rejected t=3.06853    +3.550e-180
>> retrying with dt=8.874e-181
>> > [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> > [0]PETSC ERROR: Floating point exception!
>> > [0]PETSC ERROR: Vec entry at local location 15 is not-a-number or
>> infinite at end of function: Parameter number 3!
>> >
>> > It seems it is not able to recover, it decreases the timestep so much
>> that at the end you are left with something that it not a number, or 0.
>> >
>> > Any idea how to solve that ?
>> > Some time ago you told me it could be due to a problem associated with
>> DAE formulation for boundary conditions and methods with explicit stages:
>> >
>> > Is your problem similar to ex25?  (If I remember
>> > correctly, that example uses a DAE formulation for boundary conditions
>> > and thus has trouble with some methods with explicit stages.  There may
>> > be more going on.)
>> >
>> > How can I reformulate my problem so I do not use DAE formulation for
>> boundary conditions ? For instance, if I want u=u* at i=0, or Neumann (or
>> Robin) boundary conditions ?
>> > If due to method with explicit stages, which method do you recommend ?
>> I tried the FullyImplicit option for ARKIMEX, and it's even worse, it gets
>> slower.
>> >
>> > Thanks a lot in advance for your help.
>> > Christophe
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/f36844ce/attachment.html>

From christophe.ortiz at ciemat.es  Tue Oct 22 10:36:37 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 22 Oct 2013 17:36:37 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
Message-ID: <CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>

On Tue, Oct 22, 2013 at 3:37 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 22, 2013 at 8:29 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
>
>> Hi Barry,
>>
>> Thanks for suggestions. Here is the output.
>> Before using the options you suggest I get:
>>
>>  TSAdapt 'basic': step 182 accepted t=0.341561   + 3.416e-02 wlte=1.1e-09
>> family='arkimex' scheme=0:'3' dt=3.757e-02
>>       TSAdapt 'basic': step 183 accepted t=0.375717   + 3.757e-02
>> wlte=9.05e-14 family='arkimex' scheme=0:'3' dt=4.133e-02
>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 4.133e-02
>> retrying with dt=1.033e-02
>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 1.033e-02
>> retrying with dt=2.583e-03
>> ...
>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +2.212e-198
>> retrying with dt=5.529e-199
>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +5.529e-199
>> retrying with dt=1.382e-199
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Floating point exception!
>> [0]PETSC ERROR: Vec entry at local location 12 is not-a-number or
>> infinite at end of function: Parameter number 3!
>>
>>
>>  I re-checked my Jacobian several times, and I found few errors. But now
>> I think it is ok.
>> Then, with the options you suggest, it stops immediatly. Not even one
>> timestep:
>>
>
> Something is very wrong here. The Jacobian should at least have something
> on the diagonal from the time derivative,
> but the Laplacian (u, v) and q (w) also have diagonals.
>
>    Matt
>
>
I found out that it works up to a certain time. Then, for intermediate
times, solution exhibits artificial oscillations. For larger times,
complete divergence.

In case it could matter, I start for u and v with two initial gaussian
distributions with a peak at 1e20. They are very close to surface (i=0).
For w, uniform value of 1. Do you think it could be due to a combination of
very small/very large values ?

Anyway, I tried one of the hints given in the link you sent me (Why is
Newton's method (SNES) not converging, or converges slowly?). I multiplied
all the F() and the Jacobian by hx (the mesh size) in order to rescale, as
it is suggested (*Change the units (nondimensionalization), boundary
condition scaling, or formulation so that the Jacobian is better conditioned
*). This trick is used in ex25.c but I removed it at the beginning since I
thought it was not useful...

Now it works better for larger times. The solution seems correct. However,
for longer times, it gets into trouble again. The solution presents
artificial oscillations again. And for longer times, complete divergence:

      TSAdapt 'basic': step 322 accepted t=17.8389    + 5.000e-01 wlte=
 0 family='arkimex' scheme=0:'1bee' dt=5.000e-01
    0 SNES Function norm 7.451002297320e+57
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite
at end of function: Parameter number 3!
.... ....
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecValidValues() line 30 in src/vec/vec/interface/rvector.c
[0]PETSC ERROR: PCApply() line 444 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApply() line 227 in
src/ksp/ksp/interface//home/u5751/petsc-3.4.1/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPInitialResidual() line 64 in
src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 239 in src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 441 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 220 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
[0]PETSC ERROR: main() line 457 in src/ts/examples/tutorials/diffusion.c

Other findings:

- Limiting the max timestep allowed seems helping and allows to go to
larger times without divergence.
- Increasing number of nodes seems to help too.

Does it have something to do with explicit stage of IMEX method ?


Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/c710f4f5/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Oct 22 13:07:02 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 22 Oct 2013 13:07:02 -0500
Subject: [petsc-users] SNES line search basic
In-Reply-To: <26705281.308841382426324160.JavaMail.coremail@mail.ustc.edu>
References: <26705281.308841382426324160.JavaMail.coremail@mail.ustc.edu>
Message-ID: <361D96DF-7153-4FEA-8E0E-CE219E6F122F@mcs.anl.gov>


On Oct 22, 2013, at 2:18 AM, Gong Ding <gdiso at ustc.edu> wrote:

> Hi,
> I updated petsc from 3.2 to 3.4. However, I found that the nonlinear solver previously worked well now failed with massage DIVERGED_LINE_SEARCH. 
> My problem is extremely nonlinear, cubic line search may fail but Newton method works.
> 
> With petsc 3.2, I only use basic Newton by calling 
> SNESLineSearchSet(snes, SNESLineSearchNo, PETSC_NULL) 
> With 3.4 I call basic Newton by  
> SNESLineSearchSetType(snes, SNESLINESEARCHBASIC).

   You need to use 

    SNESLineSearch ls;
    SNESGetLineSearch(snes,&ls);
    SNESLineSearchSetType(ls,SNESLINESEARCHBASIC) 

   is that what you do? 

    Try calling 

    SNESLineSearchView(ls,NULL); immediately after this and see what it reports.

   Possibly somehow later the line search is being reset back to the default.

    It is suppose to behave like previous versions of PETSc and just use the full Newton step.

   Barry

> 
> However, petsc 3.4 still reports 
> [0] SNESSolve_NEWTONLS(): fnorm=3.7500000000000006e-02, gnorm=3.7500000000000006e-02, ynorm=1.0000000000000000e+00, lssucceed=0
> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.414213562373e+00 near zero implies found a local minimum  
> 
> 
> It seems petsc 3.4 can not disable line search. Does anyone have the same problem?
> 
> Gong Ding
> 
> 
> 


From bsmith at mcs.anl.gov  Tue Oct 22 13:28:06 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 22 Oct 2013 13:28:06 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <1677196.9ZmfKVC8up@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071>
	<8D335A7E-BECF-438B-BCAC-D2960EF0DFF3@mcs.anl.gov>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
	<1677196.9ZmfKVC8up@dhcp071>
Message-ID: <579020DE-F971-4817-9FAF-760574E25B22@mcs.anl.gov>


   I have pushed a new branch to the PETSc repository called barry/reduce-dmsetup-da-memoryusage which cuts to 1/3 the amount of memory that is order dof * number of local grid points in the DMSetUp(), with this change you should see a pretty good improvement in "wasted" memory.

  See https://bitbucket.org/petsc/petsc/wiki/Home for accessing the branch.

   Barry



On Oct 22, 2013, at 3:57 AM, Juha J?ykk? <juhaj at iki.fi> wrote:

> Barry,
> 
> I seem to have touched a topic which goes way past my knowledge of PETSc 
> internals, but it's very nice to see a thorough response nevertheless. Thank 
> you. And Matthew, too.
> 
> After reading your suspicions about number of ranks, I tried with 1, 2 and 4 
> and the memory use indeed seems to go down from 1:
> 
> juhaj at dhcp071> CMD='import helpers; procdata=helpers._ProcessMemoryInfoProc(); 
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; from petsc4py 
> import PETSc; procdata=helpers._ProcessMemoryInfoProc(); print 
> procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]; da = 
> PETSc.DA().create(sizes=[100,100,100], 
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE], boundary_type=[3,0,0], 
> stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1, 
> comm=PETSc.COMM_WORLD); procdata=helpers._ProcessMemoryInfoProc(); print 
> procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]'
> juhaj at dhcp071> mpirun -np 1 python -c "$CMD"
> 21 MiB / 22280 kB
> 21 MiB / 22304 kB
> 354 MiB / 419176 kB
> juhaj at dhcp071> mpirun -np 2 python -c "$CMD"
> 22 MiB / 23276 kB
> 22 MiB / 23020 kB
> 22 MiB / 23300 kB
> 22 MiB / 23044 kB
> 141 MiB / 145324 kB
> 141 MiB / 145068 kB
> juhaj at dhcp071> mpirun -np 4 python -c "$CMD"
> 22 MiB / 23292 kB
> 22 MiB / 23036 kB
> 22 MiB / 23316 kB
> 22 MiB / 23060 kB
> 22 MiB / 23316 kB
> 22 MiB / 23340 kB
> 22 MiB / 23044 kB
> 22 MiB / 23068 kB
> 81 MiB / 83716 kB
> 82 MiB / 83976 kB
> 81 MiB / 83964 kB
> 81 MiB / 83724 kB
> 
> As one would expect, 4 ranks needs more memory than 2 ranks, but quite 
> unexpectedly, 1 rank needs more than 2! I guess you are right: the 1-rank-case 
> is not optimised and quite frankly, I don't mind: I only ever run small tests 
> with one rank. Unfortunately, trying to create the simplest possible scenario 
> to illustrate my point, I used a small DA and just one rank, precisely to 
> avoid the case where the excess memory would be due to MPI buffers or such. 
> Looks like my plan backfired. ;)
> 
> But even still, my 53 MiB lattice, without any vectors created, takes 280 or 
> 320 MiB of memory ? down to <6 from the original 6.6.
> 
> I will test with 3.3 later today if I have the time, but I'm pretty sure 
> things were "better" there.
> 
> On Monday 21 October 2013 15:23:01 Barry Smith wrote:
>>   Matt,
>> 
>>     I think you are running on 1 process where the DMDA doesn't have an
>> optimized path, when I run on 2 processes the numbers indicate nothing
>> proportional to dof* number of local points
>> 
>> dof = 12
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
>> VecScatter [0] 7 21344 VecScatterCreate()
>> [0] 2 32 VecScatterCreateCommon_PtoS()
>> [0] 39 182480 VecScatterCreate_PtoS()
>> 
>> dof = 8
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
>> VecScatter [0] 7 21344 VecScatterCreate()
>> [0] 2 32 VecScatterCreateCommon_PtoS()
>> [0] 39 176080 VecScatterCreate_PtoS()
>> 
>> dof = 4
>> 
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
>> VecScatter [0] 7 21344 VecScatterCreate()
>> [0] 2 32 VecScatterCreateCommon_PtoS()
>> [0] 39 169680 VecScatterCreate_PtoS()
>> 
>> dof = 2
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
>> VecScatter [0] 7 21344 VecScatterCreate()
>> [0] 2 32 VecScatterCreateCommon_PtoS()
>> [0] 39 166480 VecScatterCreate_PtoS()
>> 
>> dof =2 grid is 50 by 50 instead of 100 by 100
>> 
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep
>> VecScatter [0] 7 6352 VecScatterCreate()
>> [0] 2 32 VecScatterCreateCommon_PtoS()
>> [0] 39 43952 VecScatterCreate_PtoS()
>> 
>> The IS creation in the DMDA is far more troubling
>> 
>> /Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
>> 
>> dof = 2
>> 
>> [0] 1 20400 ISBlockSetIndices_Block()
>> [0] 15 3760 ISCreate()
>> [0] 4 128 ISCreate_Block()
>> [0] 1 16 ISCreate_Stride()
>> [0] 2 81600 ISGetIndices_Block()
>> [0] 1 20400 ISLocalToGlobalMappingBlock()
>> [0] 7 42016 ISLocalToGlobalMappingCreate()
>> 
>> dof = 4
>> 
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
>> [0] 1 20400 ISBlockSetIndices_Block()
>> [0] 15 3760 ISCreate()
>> [0] 4 128 ISCreate_Block()
>> [0] 1 16 ISCreate_Stride()
>> [0] 2 163200 ISGetIndices_Block()
>> [0] 1 20400 ISLocalToGlobalMappingBlock()
>> [0] 7 82816 ISLocalToGlobalMappingCreate()
>> 
>> dof = 8
>> 
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
>> [0] 1 20400 ISBlockSetIndices_Block()
>> [0] 15 3760 ISCreate()
>> [0] 4 128 ISCreate_Block()
>> [0] 1 16 ISCreate_Stride()
>> [0] 2 326400 ISGetIndices_Block()
>> [0] 1 20400 ISLocalToGlobalMappingBlock()
>> [0] 7 164416 ISLocalToGlobalMappingCreate()
>> 
>> dof = 12
>> ~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
>> [0] 1 20400 ISBlockSetIndices_Block()
>> [0] 15 3760 ISCreate()
>> [0] 4 128 ISCreate_Block()
>> [0] 1 16 ISCreate_Stride()
>> [0] 2 489600 ISGetIndices_Block()
>> [0] 1 20400 ISLocalToGlobalMappingBlock()
>> [0] 7 246016 ISLocalToGlobalMappingCreate()
>> 
>> Here the accessing of indices is at the point level (as well as block) and
>> hence memory usage is proportional to dof* local number of grid points. Of
>> course it is still only proportional to the vector size. There is some
>> improvement we could make it; with a lot of refactoring we can remove the
>> dof* completely, with a little refactoring we can bring it down to a single
>> dof*local number of grid points.
>> 
>>   I cannot understand why you are seeing memory usage 7 times more than a
>> vector. That seems like a lot.
>> 
>>   Barry
>> 
>> On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>  The PETSc DMDA object greedily allocates several arrays of data used to
>>>  set up the communication and other things like local to global mappings
>>>  even before you create any vectors. This is why you see this big bump
>>>  in memory usage.
>>> 
>>>  BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier;
>>>  at least we did not intend to make it worse. Are you sure it is using
>>>  more memory than in 3.3
>>> 
>>>  In order for use to decrease the memory usage of the DMDA setup it would
>>>  be helpful if we knew which objects created within it used the most
>>>  memory.  There is some sloppiness in that routine of not reusing memory
>>>  as well as could be, not sure how much difference that would make.
>>> 
>>> 
>>>  Barry
>>> 
>>> On Oct 21, 2013, at 7:02 AM, Juha J?ykk? <juhaj at iki.fi> wrote:
>>>> Dear list members,
>>>> 
>>>> I have noticed strange memory consumption after upgrading to 3.4 series.
>>>> I
>>>> never had time to properly investigate, but here is what happens [yes,
>>>> this
>>>> might be a petsc4py issue, but I doubt it] is
>>>> 
>>>> # helpers contains _ProcessMemoryInfoProc routine which just digs the
>>>> memory # usage data from /proc
>>>> import helpers
>>>> procdata=helpers._ProcessMemoryInfoProc()
>>>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>>>> from petsc4py import PETSc
>>>> procdata=helpers._ProcessMemoryInfoProc()
>>>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>>>> da = PETSc.DA().create(sizes=[100,100,100],
>>>> 
>>>>                     proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
>>>>                     boundary_type=[3,0,0],
>>>>                     stencil_type=PETSc.DA.StencilType.BOX,
>>>>                     dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
>>>> 
>>>> procdata=helpers._ProcessMemoryInfoProc()
>>>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>>>> vec=da.createGlobalVec()
>>>> procdata=helpers._ProcessMemoryInfoProc()
>>>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>>>> 
>>>> outputs
>>>> 
>>>> 48 MiB / 49348 kB
>>>> 48 MiB / 49360 kB
>>>> 381 MiB / 446228 kB
>>>> 435 MiB / 446228 kB
>>>> 
>>>> Which is odd: size of the actual data to be stored in the da is just
>>>> about 56 megabytes, so why does creating the da consume 7 times that?
>>>> And why does the DA reserve the memory in the first place? I thought
>>>> memory only gets allocated once an associated vector is created and it
>>>> indeed looks like the
>>>> createGlobalVec call does indeed allocate the right amount of data. But
>>>> what is that 330 MiB that DA().create() consumes? [It's actually the
>>>> .setUp() method that does the consuming, but that's not of much use as
>>>> it needs to be called before a vector can be created.]
>>>> 
>>>> Cheers,
>>>> Juha


From pengxwang at hotmail.com  Tue Oct 22 13:58:57 2013
From: pengxwang at hotmail.com (Roc Wang)
Date: Tue, 22 Oct 2013 13:58:57 -0500
Subject: [petsc-users] selection of KSP solver
Message-ID: <BAY179-W6864FAC5EAA3B01C2D7FCCB4020@phx.gbl>

Hi,

I have a linear system with about 161^3=4,173,281 unknowns.  I used the default KSP solver(to my understand it is GMRES) to solve it. The relative tolerance is set as 1.266e-9. The number of iterations is 5373 and the computing time problem is about 23.65 mins with 128 processes.   With the same problem, but I selected a pre-conditioner such as hypre, the number of iterations can be reduced dramatically, however, the computing time increases somewhat.

May someone give me some suggestions on selection of KSP solver to solve a large scale linear system faster?  Thanks a lot.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/8462db2b/attachment.html>

From knepley at gmail.com  Tue Oct 22 14:16:31 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 14:16:31 -0500
Subject: [petsc-users] selection of KSP solver
In-Reply-To: <BAY179-W6864FAC5EAA3B01C2D7FCCB4020@phx.gbl>
References: <BAY179-W6864FAC5EAA3B01C2D7FCCB4020@phx.gbl>
Message-ID: <CAMYG4Gk+BN13p3MD_-OBrawkhbdP0Tz8cGDPONneddef_H=iwg@mail.gmail.com>

On Tue, Oct 22, 2013 at 1:58 PM, Roc Wang <pengxwang at hotmail.com> wrote:

> Hi,
>
> I have a linear system with about 161^3=4,173,281 unknowns.  I used the
> default KSP solver(to my understand it is GMRES) to solve it. The relative
> tolerance is set as 1.266e-9. The number of iterations is 5373 and the
> computing time problem is about 23.65 mins with 128 processes.   With the
> same problem, but I selected a pre-conditioner such as hypre, the number of
> iterations can be reduced dramatically, however, the computing time
> increases somewhat.
>
> May someone give me some suggestions on selection of KSP solver to solve a
> large scale linear system faster?  Thanks a lot.
>

There are no blackbox preconditioners. You must find one that matches your
system. The right thing to do
is to lookup the literature on your problem and see what other people have
done. It should be easy to
replicate in PETSc.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/44c2fe75/attachment.html>

From u.tabak at tudelft.nl  Tue Oct 22 14:40:59 2013
From: u.tabak at tudelft.nl (Umut Tabak)
Date: Tue, 22 Oct 2013 21:40:59 +0200
Subject: [petsc-users] selection of KSP solver
In-Reply-To: <CAMYG4Gk+BN13p3MD_-OBrawkhbdP0Tz8cGDPONneddef_H=iwg@mail.gmail.com>
References: <BAY179-W6864FAC5EAA3B01C2D7FCCB4020@phx.gbl>
	<CAMYG4Gk+BN13p3MD_-OBrawkhbdP0Tz8cGDPONneddef_H=iwg@mail.gmail.com>
Message-ID: <5266D4CB.1090903@tudelft.nl>

On 10/22/2013 09:16 PM, Matthew Knepley wrote:
>
> There are no blackbox preconditioners. You must find one that matches 
> your system. The right thing to do
> is to lookup the literature on your problem and see what other people 
> have done. It should be easy to
> replicate in PETSc.
+ The first thing to investigate is the nature of the matrices, is that 
symmetric or not since you use GMRES I assume it is non-symmetric.
+ Second, do you have any ideas on the distribution of the 
eigenvalues(or values of the smallest and largest singular values)
+ If the problem is ill-conditioned(of course this is from personal 
experience), leave iterative methods as soon as possible and stick to 
direct solver unless you can find a well performing preconditioner(which 
is again from experience, very difficult to find).
+ Iterative methods really require one to know what he/she is doing and 
there is a vast literature on that which I find difficult to follow from 
time to time as an engineer.
+ However for well conditioned problems, their performance is amazing, 
although almost all the practical engineering problems are 
ill-conditioned. Moreover the picture gets even worse if two or three 
physics are coupled(the issue of scaling between different field variables).

These are the points from my experience.
Good luck.
Best,
Umut
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/abbec5f0/attachment.html>

From paulhuaizhang at gmail.com  Tue Oct 22 15:04:19 2013
From: paulhuaizhang at gmail.com (huaibao zhang)
Date: Tue, 22 Oct 2013 16:04:19 -0400
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
Message-ID: <74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>

Thanks for the answer. It makes sense. 

However, in my case, matrix A is huge and rather sparse, which also owns a pretty good diagonal structure although there are some other elements are nonzero. I have to  look for a better way to solve the system more efficiently. If in parallel, it is even better. 

Attached is an example for A's structure. The pink block is a matrix with 10x10 elements. The row or column in my case can be in million size. 

Thanks again. 
Paul




--
Huaibao (Paul) Zhang
Gas Surface Interactions Lab
Department of Mechanical Engineering
University of Kentucky,
Lexington, KY, 40506-0503
Office: 216 Ralph G. Anderson Building
Web:gsil.engineering.uky.edu

On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com> wrote:
> Hi Jed,
> 
> Thanks a lot for your answer. It really helps. I built parts of the matrix on each processor, then collected them into a global one according to their global position. Actually I used two MPI function instead of the one in the example, where the local size, as well as the global size is given.
> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right? 
> 
> My continuing question is since the iteration for the system is global. Is it more efficient if I solve locally instead. ie. solve parts on each of the processor instead of doing globally. 
> 
> No, because this ignores the coupling between domains.
> 
>   Matt
>  
> Thanks again,
> 
> Paul
> 
> 
> 
> 
> 
> 
> 
> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> paul zhang <paulhuaizhang at gmail.com> writes:
> 
> > I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
> > system. Here is what I am doing. I cut my computation domain into many
> > pieces, in each of them I compute independently by solving fluid equations.
> > This has nothing to do with PETSc. Finally, I collect all of the
> > information and load it to a whole A matrix.
> 
> I hope you build parts of this matrix on each processor, as is done in
> the examples.  Note the range Istart to Iend here:
> 
> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
> 
> > My question is how PETSc functions work in parallel in my case. There are
> > two guesses to me. First, PETSc solves its own matrix for each domain using
> > local processor, although A is a global. For the values like number of
> > iterations, solution vector, their numbers should have equaled to the
> > number of processors I applied, but I get only one value for each of them.
> > The reason is that the processors must talk with each other once all of
> > their work is done, that is why I received the "all reduced" value. This is
> > more logical than my second guess.
> 
> It does not work because the solution operators are global, so to solve
> the problem, the iteration must be global.
> 
> > In the second one, the system is solved in parallel too. But PETSc function
> > redistributes the global sparse matrix A to each of the processors after
> > its load is complete. That is to say now each processor may not solve the
> > its own partition matrix.
> 
> Hopefully you build the matrix already-distributed.  The default
> _preconditioner_ is local, but the iteration is global.  PETSc does not
> "redistribute" the matrix automatically, though if you call
> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
> choose them.
> 
> 
> 
> -- 
> Huaibao (Paul) Zhang
> Gas Surface Interactions Lab
> Department of Mechanical Engineering
> University of Kentucky,
> Lexington,
> KY, 40506-0503
> Office: 216 Ralph G. Anderson Building
> Web:gsil.engineering.uky.edu
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/8a4d370b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/8a4d370b/attachment-0001.png>

From knepley at gmail.com  Tue Oct 22 15:08:16 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 15:08:16 -0500
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
Message-ID: <CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>

On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com>wrote:

> Thanks for the answer. It makes sense.
>
> However, in my case, matrix A is huge and rather sparse, which also owns a
> pretty good diagonal structure although there are some other elements are
> nonzero. I have to  look for a better way to solve the system more
> efficiently. If in parallel, it is even better.
>
> Attached is an example for A's structure. The pink block is a matrix with
> 10x10 elements. The row or column in my case can be in million size.
>

The analytic character of the operator is usually more important than the
sparsity structure for scalable solvers.
The pattern matters a lot for direct solvers, and you should definitely try
them (SuperLU_dist or MUMPS in PETSc).
If they use too much memory or are too slow, then you need to investigate
good preconditioners for iterative methods.

   Matt


> Thanks again.
> Paul
>
>
>
> --
> Huaibao (Paul) Zhang
> *Gas Surface Interactions Lab*
> Department of Mechanical Engineering
> University of Kentucky,
> Lexington, KY, 40506-0503
> *Office*: 216 Ralph G. Anderson Building
> *Web*:gsil.engineering.uky.edu
>
> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>
>> Hi Jed,
>>
>> Thanks a lot for your answer. It really helps. I built parts of the
>> matrix on each processor, then collected them into a global one according
>> to their global position. Actually I used two MPI function instead of the
>> one in the example, where the local size, as well as the global size is
>> given.
>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>
>> My continuing question is since the iteration for the system is global.
>> Is it more efficient if I solve locally instead. ie. solve parts on each of
>> the processor instead of doing globally.
>>
>
> No, because this ignores the coupling between domains.
>
>   Matt
>
>
>> Thanks again,
>>
>> Paul
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>
>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>
>>> > I am using KSP, more specifically FGMRES method, with MPI to solve Ax=b
>>> > system. Here is what I am doing. I cut my computation domain into many
>>> > pieces, in each of them I compute independently by solving fluid
>>> equations.
>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>> > information and load it to a whole A matrix.
>>>
>>> I hope you build parts of this matrix on each processor, as is done in
>>> the examples.  Note the range Istart to Iend here:
>>>
>>>
>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>
>>> > My question is how PETSc functions work in parallel in my case. There
>>> are
>>> > two guesses to me. First, PETSc solves its own matrix for each domain
>>> using
>>> > local processor, although A is a global. For the values like number of
>>> > iterations, solution vector, their numbers should have equaled to the
>>> > number of processors I applied, but I get only one value for each of
>>> them.
>>> > The reason is that the processors must talk with each other once all of
>>> > their work is done, that is why I received the "all reduced" value.
>>> This is
>>> > more logical than my second guess.
>>>
>>> It does not work because the solution operators are global, so to solve
>>> the problem, the iteration must be global.
>>>
>>> > In the second one, the system is solved in parallel too. But PETSc
>>> function
>>> > redistributes the global sparse matrix A to each of the processors
>>> after
>>> > its load is complete. That is to say now each processor may not solve
>>> the
>>> > its own partition matrix.
>>>
>>> Hopefully you build the matrix already-distributed.  The default
>>> _preconditioner_ is local, but the iteration is global.  PETSc does not
>>> "redistribute" the matrix automatically, though if you call
>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>> choose them.
>>>
>>
>>
>>
>> --
>> Huaibao (Paul) Zhang
>> *Gas Surface Interactions Lab*
>> Department of Mechanical Engineering
>> University of Kentucky,
>> Lexington,
>> KY, 40506-0503*
>> Office*: 216 Ralph G. Anderson Building
>> *Web*:gsil.engineering.uky.edu
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/40ae0d9d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/40ae0d9d/attachment.png>

From paulhuaizhang at gmail.com  Tue Oct 22 15:12:49 2013
From: paulhuaizhang at gmail.com (paul zhang)
Date: Tue, 22 Oct 2013 16:12:49 -0400
Subject: [petsc-users] Fwd:  Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
	<CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>
	<CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
Message-ID: <CAA4k2zLmfX-E_o78dFFtbsDSNp1qV+vCHT-RRfjzVCSo7RPTeg@mail.gmail.com>

---------- Forwarded message ----------
From: paul zhang <paulhuaizhang at gmail.com>
Date: Tue, Oct 22, 2013 at 4:12 PM
Subject: Re: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
To: Matthew Knepley <knepley at gmail.com>


One more question, can I solve the system in parallel?




On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com>wrote:
>
>> Thanks for the answer. It makes sense.
>>
>> However, in my case, matrix A is huge and rather sparse, which also owns
>> a pretty good diagonal structure although there are some other elements are
>> nonzero. I have to  look for a better way to solve the system more
>> efficiently. If in parallel, it is even better.
>>
>> Attached is an example for A's structure. The pink block is a matrix with
>> 10x10 elements. The row or column in my case can be in million size.
>>
>
> The analytic character of the operator is usually more important than the
> sparsity structure for scalable solvers.
> The pattern matters a lot for direct solvers, and you should definitely
> try them (SuperLU_dist or MUMPS in PETSc).
> If they use too much memory or are too slow, then you need to investigate
> good preconditioners for iterative methods.
>
>    Matt
>
>
>> Thanks again.
>> Paul
>>
>>
>>
>>  --
>> Huaibao (Paul) Zhang
>> *Gas Surface Interactions Lab*
>> Department of Mechanical Engineering
>> University of Kentucky,
>> Lexington, KY, 40506-0503
>> *Office*: 216 Ralph G. Anderson Building
>> *Web*:gsil.engineering.uky.edu
>>
>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>
>>> Hi Jed,
>>>
>>> Thanks a lot for your answer. It really helps. I built parts of the
>>> matrix on each processor, then collected them into a global one according
>>> to their global position. Actually I used two MPI function instead of the
>>> one in the example, where the local size, as well as the global size is
>>> given.
>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>
>>> My continuing question is since the iteration for the system is global.
>>> Is it more efficient if I solve locally instead. ie. solve parts on each of
>>> the processor instead of doing globally.
>>>
>>
>> No, because this ignores the coupling between domains.
>>
>>   Matt
>>
>>
>>> Thanks again,
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>
>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>
>>>> > I am using KSP, more specifically FGMRES method, with MPI to solve
>>>> Ax=b
>>>> > system. Here is what I am doing. I cut my computation domain into many
>>>> > pieces, in each of them I compute independently by solving fluid
>>>> equations.
>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>> > information and load it to a whole A matrix.
>>>>
>>>> I hope you build parts of this matrix on each processor, as is done in
>>>> the examples.  Note the range Istart to Iend here:
>>>>
>>>>
>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>
>>>> > My question is how PETSc functions work in parallel in my case. There
>>>> are
>>>> > two guesses to me. First, PETSc solves its own matrix for each domain
>>>> using
>>>> > local processor, although A is a global. For the values like number of
>>>> > iterations, solution vector, their numbers should have equaled to the
>>>> > number of processors I applied, but I get only one value for each of
>>>> them.
>>>> > The reason is that the processors must talk with each other once all
>>>> of
>>>> > their work is done, that is why I received the "all reduced" value.
>>>> This is
>>>> > more logical than my second guess.
>>>>
>>>> It does not work because the solution operators are global, so to solve
>>>> the problem, the iteration must be global.
>>>>
>>>> > In the second one, the system is solved in parallel too. But PETSc
>>>> function
>>>> > redistributes the global sparse matrix A to each of the processors
>>>> after
>>>> > its load is complete. That is to say now each processor may not solve
>>>> the
>>>> > its own partition matrix.
>>>>
>>>> Hopefully you build the matrix already-distributed.  The default
>>>> _preconditioner_ is local, but the iteration is global.  PETSc does not
>>>> "redistribute" the matrix automatically, though if you call
>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>> choose them.
>>>>
>>>
>>>
>>>
>>> --
>>> Huaibao (Paul) Zhang
>>> *Gas Surface Interactions Lab*
>>>  Department of Mechanical Engineering
>>> University of Kentucky,
>>> Lexington,
>>> KY, 40506-0503*
>>> Office*: 216 Ralph G. Anderson Building
>>> *Web*:gsil.engineering.uky.edu
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/9d513330/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/9d513330/attachment-0001.png>

From knepley at gmail.com  Tue Oct 22 15:14:59 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 15:14:59 -0500
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
	<CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>
	<CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
Message-ID: <CAMYG4GmrcND79aShUYr_g67=YgS=_XVY=MYx5PXXdGWLPsovMw@mail.gmail.com>

On Tue, Oct 22, 2013 at 3:12 PM, paul zhang <paulhuaizhang at gmail.com> wrote:

> One more question, can I solve the system in parallel?
>

Yes, or you would be using ETSC :)

   Matt


>
> On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com>wrote:
>>
>>> Thanks for the answer. It makes sense.
>>>
>>> However, in my case, matrix A is huge and rather sparse, which also owns
>>> a pretty good diagonal structure although there are some other elements are
>>> nonzero. I have to  look for a better way to solve the system more
>>> efficiently. If in parallel, it is even better.
>>>
>>> Attached is an example for A's structure. The pink block is a matrix
>>> with 10x10 elements. The row or column in my case can be in million size.
>>>
>>
>> The analytic character of the operator is usually more important than the
>> sparsity structure for scalable solvers.
>> The pattern matters a lot for direct solvers, and you should definitely
>> try them (SuperLU_dist or MUMPS in PETSc).
>> If they use too much memory or are too slow, then you need to investigate
>> good preconditioners for iterative methods.
>>
>>    Matt
>>
>>
>>> Thanks again.
>>> Paul
>>>
>>>
>>>
>>>  --
>>> Huaibao (Paul) Zhang
>>> *Gas Surface Interactions Lab*
>>> Department of Mechanical Engineering
>>> University of Kentucky,
>>> Lexington, KY, 40506-0503
>>> *Office*: 216 Ralph G. Anderson Building
>>> *Web*:gsil.engineering.uky.edu
>>>
>>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>>
>>>> Hi Jed,
>>>>
>>>> Thanks a lot for your answer. It really helps. I built parts of the
>>>> matrix on each processor, then collected them into a global one according
>>>> to their global position. Actually I used two MPI function instead of the
>>>> one in the example, where the local size, as well as the global size is
>>>> given.
>>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>>
>>>> My continuing question is since the iteration for the system is global.
>>>> Is it more efficient if I solve locally instead. ie. solve parts on each of
>>>> the processor instead of doing globally.
>>>>
>>>
>>> No, because this ignores the coupling between domains.
>>>
>>>   Matt
>>>
>>>
>>>> Thanks again,
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>
>>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>>
>>>>> > I am using KSP, more specifically FGMRES method, with MPI to solve
>>>>> Ax=b
>>>>> > system. Here is what I am doing. I cut my computation domain into
>>>>> many
>>>>> > pieces, in each of them I compute independently by solving fluid
>>>>> equations.
>>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>>> > information and load it to a whole A matrix.
>>>>>
>>>>> I hope you build parts of this matrix on each processor, as is done in
>>>>> the examples.  Note the range Istart to Iend here:
>>>>>
>>>>>
>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>>
>>>>> > My question is how PETSc functions work in parallel in my case.
>>>>> There are
>>>>> > two guesses to me. First, PETSc solves its own matrix for each
>>>>> domain using
>>>>> > local processor, although A is a global. For the values like number
>>>>> of
>>>>> > iterations, solution vector, their numbers should have equaled to the
>>>>> > number of processors I applied, but I get only one value for each of
>>>>> them.
>>>>> > The reason is that the processors must talk with each other once all
>>>>> of
>>>>> > their work is done, that is why I received the "all reduced" value.
>>>>> This is
>>>>> > more logical than my second guess.
>>>>>
>>>>> It does not work because the solution operators are global, so to solve
>>>>> the problem, the iteration must be global.
>>>>>
>>>>> > In the second one, the system is solved in parallel too. But PETSc
>>>>> function
>>>>> > redistributes the global sparse matrix A to each of the processors
>>>>> after
>>>>> > its load is complete. That is to say now each processor may not
>>>>> solve the
>>>>> > its own partition matrix.
>>>>>
>>>>> Hopefully you build the matrix already-distributed.  The default
>>>>> _preconditioner_ is local, but the iteration is global.  PETSc does not
>>>>> "redistribute" the matrix automatically, though if you call
>>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>>> choose them.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Huaibao (Paul) Zhang
>>>> *Gas Surface Interactions Lab*
>>>>  Department of Mechanical Engineering
>>>> University of Kentucky,
>>>> Lexington,
>>>> KY, 40506-0503*
>>>> Office*: 216 Ralph G. Anderson Building
>>>> *Web*:gsil.engineering.uky.edu
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> Huaibao (Paul) Zhang
> *Gas Surface Interactions Lab*
> Department of Mechanical Engineering
> University of Kentucky,
> Lexington,
> KY, 40506-0503*
> Office*: 216 Ralph G. Anderson Building
> *Web*:gsil.engineering.uky.edu
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/f650950e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/f650950e/attachment-0001.png>

From paulhuaizhang at gmail.com  Tue Oct 22 15:16:38 2013
From: paulhuaizhang at gmail.com (paul zhang)
Date: Tue, 22 Oct 2013 16:16:38 -0400
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAMYG4GmrcND79aShUYr_g67=YgS=_XVY=MYx5PXXdGWLPsovMw@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
	<CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>
	<CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
	<CAMYG4GmrcND79aShUYr_g67=YgS=_XVY=MYx5PXXdGWLPsovMw@mail.gmail.com>
Message-ID: <CAA4k2zLA-GdB28JF9vTMmLMoP0o7O5jhfsV1a2vh_9qSpyUWVQ@mail.gmail.com>

That is a good one. I mean for my case. which method can I try?


On Tue, Oct 22, 2013 at 4:14 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 22, 2013 at 3:12 PM, paul zhang <paulhuaizhang at gmail.com>wrote:
>
>> One more question, can I solve the system in parallel?
>>
>
> Yes, or you would be using ETSC :)
>
>    Matt
>
>
>>
>> On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com>wrote:
>>>
>>>> Thanks for the answer. It makes sense.
>>>>
>>>> However, in my case, matrix A is huge and rather sparse, which also
>>>> owns a pretty good diagonal structure although there are some other
>>>> elements are nonzero. I have to  look for a better way to solve the system
>>>> more efficiently. If in parallel, it is even better.
>>>>
>>>> Attached is an example for A's structure. The pink block is a matrix
>>>> with 10x10 elements. The row or column in my case can be in million size.
>>>>
>>>
>>> The analytic character of the operator is usually more important than
>>> the sparsity structure for scalable solvers.
>>> The pattern matters a lot for direct solvers, and you should definitely
>>> try them (SuperLU_dist or MUMPS in PETSc).
>>> If they use too much memory or are too slow, then you need to
>>> investigate good preconditioners for iterative methods.
>>>
>>>    Matt
>>>
>>>
>>>> Thanks again.
>>>> Paul
>>>>
>>>>
>>>>
>>>>  --
>>>> Huaibao (Paul) Zhang
>>>> *Gas Surface Interactions Lab*
>>>> Department of Mechanical Engineering
>>>> University of Kentucky,
>>>> Lexington, KY, 40506-0503
>>>> *Office*: 216 Ralph G. Anderson Building
>>>> *Web*:gsil.engineering.uky.edu
>>>>
>>>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>>>
>>>>> Hi Jed,
>>>>>
>>>>> Thanks a lot for your answer. It really helps. I built parts of the
>>>>> matrix on each processor, then collected them into a global one according
>>>>> to their global position. Actually I used two MPI function instead of the
>>>>> one in the example, where the local size, as well as the global size is
>>>>> given.
>>>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>>>
>>>>> My continuing question is since the iteration for the system is
>>>>> global. Is it more efficient if I solve locally instead. ie. solve parts on
>>>>> each of the processor instead of doing globally.
>>>>>
>>>>
>>>> No, because this ignores the coupling between domains.
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> Thanks again,
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>
>>>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>>>
>>>>>> > I am using KSP, more specifically FGMRES method, with MPI to solve
>>>>>> Ax=b
>>>>>> > system. Here is what I am doing. I cut my computation domain into
>>>>>> many
>>>>>> > pieces, in each of them I compute independently by solving fluid
>>>>>> equations.
>>>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>>>> > information and load it to a whole A matrix.
>>>>>>
>>>>>> I hope you build parts of this matrix on each processor, as is done in
>>>>>> the examples.  Note the range Istart to Iend here:
>>>>>>
>>>>>>
>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>>>
>>>>>> > My question is how PETSc functions work in parallel in my case.
>>>>>> There are
>>>>>> > two guesses to me. First, PETSc solves its own matrix for each
>>>>>> domain using
>>>>>> > local processor, although A is a global. For the values like number
>>>>>> of
>>>>>> > iterations, solution vector, their numbers should have equaled to
>>>>>> the
>>>>>> > number of processors I applied, but I get only one value for each
>>>>>> of them.
>>>>>> > The reason is that the processors must talk with each other once
>>>>>> all of
>>>>>> > their work is done, that is why I received the "all reduced" value.
>>>>>> This is
>>>>>> > more logical than my second guess.
>>>>>>
>>>>>> It does not work because the solution operators are global, so to
>>>>>> solve
>>>>>> the problem, the iteration must be global.
>>>>>>
>>>>>> > In the second one, the system is solved in parallel too. But PETSc
>>>>>> function
>>>>>> > redistributes the global sparse matrix A to each of the processors
>>>>>> after
>>>>>> > its load is complete. That is to say now each processor may not
>>>>>> solve the
>>>>>> > its own partition matrix.
>>>>>>
>>>>>> Hopefully you build the matrix already-distributed.  The default
>>>>>> _preconditioner_ is local, but the iteration is global.  PETSc does
>>>>>> not
>>>>>> "redistribute" the matrix automatically, though if you call
>>>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>>>> choose them.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Huaibao (Paul) Zhang
>>>>> *Gas Surface Interactions Lab*
>>>>>  Department of Mechanical Engineering
>>>>> University of Kentucky,
>>>>> Lexington,
>>>>> KY, 40506-0503*
>>>>> Office*: 216 Ralph G. Anderson Building
>>>>> *Web*:gsil.engineering.uky.edu
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Huaibao (Paul) Zhang
>> *Gas Surface Interactions Lab*
>> Department of Mechanical Engineering
>> University of Kentucky,
>> Lexington,
>> KY, 40506-0503*
>> Office*: 216 Ralph G. Anderson Building
>> *Web*:gsil.engineering.uky.edu
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/5ca7fd58/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/5ca7fd58/attachment-0001.png>

From knepley at gmail.com  Tue Oct 22 15:25:07 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 Oct 2013 15:25:07 -0500
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAA4k2zLA-GdB28JF9vTMmLMoP0o7O5jhfsV1a2vh_9qSpyUWVQ@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
	<CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>
	<CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
	<CAMYG4GmrcND79aShUYr_g67=YgS=_XVY=MYx5PXXdGWLPsovMw@mail.gmail.com>
	<CAA4k2zLA-GdB28JF9vTMmLMoP0o7O5jhfsV1a2vh_9qSpyUWVQ@mail.gmail.com>
Message-ID: <CAMYG4G=SQkNK671zqTm=utEbosJLgbzGvDGSBWV_iHiNYYGBCQ@mail.gmail.com>

On Tue, Oct 22, 2013 at 3:16 PM, paul zhang <paulhuaizhang at gmail.com> wrote:

> That is a good one. I mean for my case. which method can I try?
>

Both those direct methods are parallel (SuperLU_dist and MUMPS).

  Matt


> On Tue, Oct 22, 2013 at 4:14 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Oct 22, 2013 at 3:12 PM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>
>>> One more question, can I solve the system in parallel?
>>>
>>
>> Yes, or you would be using ETSC :)
>>
>>    Matt
>>
>>
>>>
>>> On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <paulhuaizhang at gmail.com
>>>> > wrote:
>>>>
>>>>> Thanks for the answer. It makes sense.
>>>>>
>>>>> However, in my case, matrix A is huge and rather sparse, which also
>>>>> owns a pretty good diagonal structure although there are some other
>>>>> elements are nonzero. I have to  look for a better way to solve the system
>>>>> more efficiently. If in parallel, it is even better.
>>>>>
>>>>> Attached is an example for A's structure. The pink block is a matrix
>>>>> with 10x10 elements. The row or column in my case can be in million size.
>>>>>
>>>>
>>>> The analytic character of the operator is usually more important than
>>>> the sparsity structure for scalable solvers.
>>>> The pattern matters a lot for direct solvers, and you should definitely
>>>> try them (SuperLU_dist or MUMPS in PETSc).
>>>> If they use too much memory or are too slow, then you need to
>>>> investigate good preconditioners for iterative methods.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> Thanks again.
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> Huaibao (Paul) Zhang
>>>>> *Gas Surface Interactions Lab*
>>>>> Department of Mechanical Engineering
>>>>> University of Kentucky,
>>>>> Lexington, KY, 40506-0503
>>>>> *Office*: 216 Ralph G. Anderson Building
>>>>> *Web*:gsil.engineering.uky.edu
>>>>>
>>>>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>>>>
>>>>>> Hi Jed,
>>>>>>
>>>>>> Thanks a lot for your answer. It really helps. I built parts of the
>>>>>> matrix on each processor, then collected them into a global one according
>>>>>> to their global position. Actually I used two MPI function instead of the
>>>>>> one in the example, where the local size, as well as the global size is
>>>>>> given.
>>>>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>>>>
>>>>>> My continuing question is since the iteration for the system is
>>>>>> global. Is it more efficient if I solve locally instead. ie. solve parts on
>>>>>> each of the processor instead of doing globally.
>>>>>>
>>>>>
>>>>> No, because this ignores the coupling between domains.
>>>>>
>>>>>   Matt
>>>>>
>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>>
>>>>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>>>>
>>>>>>> > I am using KSP, more specifically FGMRES method, with MPI to solve
>>>>>>> Ax=b
>>>>>>> > system. Here is what I am doing. I cut my computation domain into
>>>>>>> many
>>>>>>> > pieces, in each of them I compute independently by solving fluid
>>>>>>> equations.
>>>>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>>>>> > information and load it to a whole A matrix.
>>>>>>>
>>>>>>> I hope you build parts of this matrix on each processor, as is done
>>>>>>> in
>>>>>>> the examples.  Note the range Istart to Iend here:
>>>>>>>
>>>>>>>
>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>>>>
>>>>>>> > My question is how PETSc functions work in parallel in my case.
>>>>>>> There are
>>>>>>> > two guesses to me. First, PETSc solves its own matrix for each
>>>>>>> domain using
>>>>>>> > local processor, although A is a global. For the values like
>>>>>>> number of
>>>>>>> > iterations, solution vector, their numbers should have equaled to
>>>>>>> the
>>>>>>> > number of processors I applied, but I get only one value for each
>>>>>>> of them.
>>>>>>> > The reason is that the processors must talk with each other once
>>>>>>> all of
>>>>>>> > their work is done, that is why I received the "all reduced"
>>>>>>> value. This is
>>>>>>> > more logical than my second guess.
>>>>>>>
>>>>>>> It does not work because the solution operators are global, so to
>>>>>>> solve
>>>>>>> the problem, the iteration must be global.
>>>>>>>
>>>>>>> > In the second one, the system is solved in parallel too. But PETSc
>>>>>>> function
>>>>>>> > redistributes the global sparse matrix A to each of the processors
>>>>>>> after
>>>>>>> > its load is complete. That is to say now each processor may not
>>>>>>> solve the
>>>>>>> > its own partition matrix.
>>>>>>>
>>>>>>> Hopefully you build the matrix already-distributed.  The default
>>>>>>> _preconditioner_ is local, but the iteration is global.  PETSc does
>>>>>>> not
>>>>>>> "redistribute" the matrix automatically, though if you call
>>>>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>>>>> choose them.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Huaibao (Paul) Zhang
>>>>>> *Gas Surface Interactions Lab*
>>>>>>  Department of Mechanical Engineering
>>>>>> University of Kentucky,
>>>>>> Lexington,
>>>>>> KY, 40506-0503*
>>>>>> Office*: 216 Ralph G. Anderson Building
>>>>>> *Web*:gsil.engineering.uky.edu
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>>
>>> --
>>> Huaibao (Paul) Zhang
>>> *Gas Surface Interactions Lab*
>>> Department of Mechanical Engineering
>>> University of Kentucky,
>>> Lexington,
>>> KY, 40506-0503*
>>> Office*: 216 Ralph G. Anderson Building
>>> *Web*:gsil.engineering.uky.edu
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
>
> --
> Huaibao (Paul) Zhang
> *Gas Surface Interactions Lab*
> Department of Mechanical Engineering
> University of Kentucky,
> Lexington,
> KY, 40506-0503*
> Office*: 216 Ralph G. Anderson Building
> *Web*:gsil.engineering.uky.edu
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/a11a0a17/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/a11a0a17/attachment-0001.png>

From paulhuaizhang at gmail.com  Tue Oct 22 15:27:06 2013
From: paulhuaizhang at gmail.com (paul zhang)
Date: Tue, 22 Oct 2013 16:27:06 -0400
Subject: [petsc-users] Fwd: How PETSc solves Ax=b in parallel
In-Reply-To: <CAMYG4G=SQkNK671zqTm=utEbosJLgbzGvDGSBWV_iHiNYYGBCQ@mail.gmail.com>
References: <CAA4k2zJ1c_YUjxy26k+ZB9UeFpwakOK+Rn8Ptv9Z6iWk64VYbg@mail.gmail.com>
	<CAA4k2z+KOVDsAUj+mNGFX1MAF65_KPkz=A=WrVcTRkA=rPcNYg@mail.gmail.com>
	<8738nuzl3t.fsf@mcs.anl.gov>
	<CAA4k2zKFm2acQ+7GgDwA7Mwqt2ZFdipefEHapSdJnmSoSaFLEQ@mail.gmail.com>
	<CAMYG4GkTds-d6sSRnO=Rwf5SMr2EWZ2CtGBxSwQa7enkcBKDQg@mail.gmail.com>
	<74743F0E-2FAA-407A-9FF3-E930BEFFC4C3@gmail.com>
	<CAMYG4GmookK8zC2YwXv3bBzK9qMVhqLAWtasL4rLqV-MCZz8WQ@mail.gmail.com>
	<CAA4k2zKhXHt7S7oJAz2X5CvOY7BFJzCf97q2Om3SD+E_WZRF_w@mail.gmail.com>
	<CAMYG4GmrcND79aShUYr_g67=YgS=_XVY=MYx5PXXdGWLPsovMw@mail.gmail.com>
	<CAA4k2zLA-GdB28JF9vTMmLMoP0o7O5jhfsV1a2vh_9qSpyUWVQ@mail.gmail.com>
	<CAMYG4G=SQkNK671zqTm=utEbosJLgbzGvDGSBWV_iHiNYYGBCQ@mail.gmail.com>
Message-ID: <CAA4k2zKvUf66=+HZc6zNngMAyjug1xJ4FjivOT9Y-nspTGXnwQ@mail.gmail.com>

Appreciate it.
Paul


On Tue, Oct 22, 2013 at 4:25 PM, Matthew Knepley <knepley at gmail.com> wrote:

>
> On Tue, Oct 22, 2013 at 3:16 PM, paul zhang <paulhuaizhang at gmail.com>wrote:
>
>> That is a good one. I mean for my case. which method can I try?
>>
>
> Both those direct methods are parallel (SuperLU_dist and MUMPS).
>
>   Matt
>
>
>> On Tue, Oct 22, 2013 at 4:14 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Tue, Oct 22, 2013 at 3:12 PM, paul zhang <paulhuaizhang at gmail.com>wrote:
>>>
>>>> One more question, can I solve the system in parallel?
>>>>
>>>
>>> Yes, or you would be using ETSC :)
>>>
>>>    Matt
>>>
>>>
>>>>
>>>> On Tue, Oct 22, 2013 at 4:08 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Tue, Oct 22, 2013 at 3:04 PM, huaibao zhang <
>>>>> paulhuaizhang at gmail.com> wrote:
>>>>>
>>>>>> Thanks for the answer. It makes sense.
>>>>>>
>>>>>> However, in my case, matrix A is huge and rather sparse, which also
>>>>>> owns a pretty good diagonal structure although there are some other
>>>>>> elements are nonzero. I have to  look for a better way to solve the system
>>>>>> more efficiently. If in parallel, it is even better.
>>>>>>
>>>>>> Attached is an example for A's structure. The pink block is a matrix
>>>>>> with 10x10 elements. The row or column in my case can be in million size.
>>>>>>
>>>>>
>>>>> The analytic character of the operator is usually more important than
>>>>> the sparsity structure for scalable solvers.
>>>>> The pattern matters a lot for direct solvers, and you should
>>>>> definitely try them (SuperLU_dist or MUMPS in PETSc).
>>>>> If they use too much memory or are too slow, then you need to
>>>>> investigate good preconditioners for iterative methods.
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> Thanks again.
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>>
>>>>>>  --
>>>>>> Huaibao (Paul) Zhang
>>>>>> *Gas Surface Interactions Lab*
>>>>>> Department of Mechanical Engineering
>>>>>> University of Kentucky,
>>>>>> Lexington, KY, 40506-0503
>>>>>> *Office*: 216 Ralph G. Anderson Building
>>>>>> *Web*:gsil.engineering.uky.edu
>>>>>>
>>>>>> On Oct 21, 2013, at 12:53 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Mon, Oct 21, 2013 at 11:23 AM, paul zhang <paulhuaizhang at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Jed,
>>>>>>>
>>>>>>> Thanks a lot for your answer. It really helps. I built parts of the
>>>>>>> matrix on each processor, then collected them into a global one according
>>>>>>> to their global position. Actually I used two MPI function instead of the
>>>>>>> one in the example, where the local size, as well as the global size is
>>>>>>> given.
>>>>>>> VecCreateMPI and MatCreateMPIAIJ. It does not really matter right?
>>>>>>>
>>>>>>> My continuing question is since the iteration for the system is
>>>>>>> global. Is it more efficient if I solve locally instead. ie. solve parts on
>>>>>>> each of the processor instead of doing globally.
>>>>>>>
>>>>>>
>>>>>> No, because this ignores the coupling between domains.
>>>>>>
>>>>>>   Matt
>>>>>>
>>>>>>
>>>>>>> Thanks again,
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 21, 2013 at 11:42 AM, Jed Brown <jedbrown at mcs.anl.gov>wrote:
>>>>>>>
>>>>>>>> paul zhang <paulhuaizhang at gmail.com> writes:
>>>>>>>>
>>>>>>>> > I am using KSP, more specifically FGMRES method, with MPI to
>>>>>>>> solve Ax=b
>>>>>>>> > system. Here is what I am doing. I cut my computation domain into
>>>>>>>> many
>>>>>>>> > pieces, in each of them I compute independently by solving fluid
>>>>>>>> equations.
>>>>>>>> > This has nothing to do with PETSc. Finally, I collect all of the
>>>>>>>> > information and load it to a whole A matrix.
>>>>>>>>
>>>>>>>> I hope you build parts of this matrix on each processor, as is done
>>>>>>>> in
>>>>>>>> the examples.  Note the range Istart to Iend here:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex2.c.html
>>>>>>>>
>>>>>>>> > My question is how PETSc functions work in parallel in my case.
>>>>>>>> There are
>>>>>>>> > two guesses to me. First, PETSc solves its own matrix for each
>>>>>>>> domain using
>>>>>>>> > local processor, although A is a global. For the values like
>>>>>>>> number of
>>>>>>>> > iterations, solution vector, their numbers should have equaled to
>>>>>>>> the
>>>>>>>> > number of processors I applied, but I get only one value for each
>>>>>>>> of them.
>>>>>>>> > The reason is that the processors must talk with each other once
>>>>>>>> all of
>>>>>>>> > their work is done, that is why I received the "all reduced"
>>>>>>>> value. This is
>>>>>>>> > more logical than my second guess.
>>>>>>>>
>>>>>>>> It does not work because the solution operators are global, so to
>>>>>>>> solve
>>>>>>>> the problem, the iteration must be global.
>>>>>>>>
>>>>>>>> > In the second one, the system is solved in parallel too. But
>>>>>>>> PETSc function
>>>>>>>> > redistributes the global sparse matrix A to each of the
>>>>>>>> processors after
>>>>>>>> > its load is complete. That is to say now each processor may not
>>>>>>>> solve the
>>>>>>>> > its own partition matrix.
>>>>>>>>
>>>>>>>> Hopefully you build the matrix already-distributed.  The default
>>>>>>>> _preconditioner_ is local, but the iteration is global.  PETSc does
>>>>>>>> not
>>>>>>>> "redistribute" the matrix automatically, though if you call
>>>>>>>> MatSetSizes() and pass PETSC_DECIDE for the local sizes, PETSc will
>>>>>>>> choose them.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Huaibao (Paul) Zhang
>>>>>>> *Gas Surface Interactions Lab*
>>>>>>>  Department of Mechanical Engineering
>>>>>>> University of Kentucky,
>>>>>>> Lexington,
>>>>>>> KY, 40506-0503*
>>>>>>> Office*: 216 Ralph G. Anderson Building
>>>>>>> *Web*:gsil.engineering.uky.edu
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Huaibao (Paul) Zhang
>>>> *Gas Surface Interactions Lab*
>>>> Department of Mechanical Engineering
>>>> University of Kentucky,
>>>> Lexington,
>>>> KY, 40506-0503*
>>>> Office*: 216 Ralph G. Anderson Building
>>>> *Web*:gsil.engineering.uky.edu
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> Huaibao (Paul) Zhang
>> *Gas Surface Interactions Lab*
>> Department of Mechanical Engineering
>> University of Kentucky,
>> Lexington,
>> KY, 40506-0503*
>> Office*: 216 Ralph G. Anderson Building
>> *Web*:gsil.engineering.uky.edu
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Huaibao (Paul) Zhang
*Gas Surface Interactions Lab*
Department of Mechanical Engineering
University of Kentucky,
Lexington,
KY, 40506-0503*
Office*: 216 Ralph G. Anderson Building
*Web*:gsil.engineering.uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/b681df09/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 12602 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131022/b681df09/attachment-0001.png>

From gdiso at ustc.edu  Tue Oct 22 23:04:12 2013
From: gdiso at ustc.edu (Gong Ding)
Date: Wed, 23 Oct 2013 12:04:12 +0800 (CST)
Subject: [petsc-users] SNES line search basic
Message-ID: <10743052.310031382501052921.JavaMail.coremail@mail.ustc.edu>

Hi all,
The problem is solved.
I previously use

ierr = SNESSetType(snes,SNESNEWTONLS);
ierr = SNESLineSearchCreate(PETSC_COMM_WORLD, &snesls);
ierr = SNESLineSearchSetType(snesls, SNESLINESEARCHBASIC);
This method create a new snesls, but not work.

 
Barry told me that I should use 
ierr = SNESGetLineSearch(snes, &snesls);
instead of 
ierr = SNESLineSearchCreate(PETSC_COMM_WORLD, &snesls);

Then everything is ok now.

Thanks very much.

Gong Ding
      


> Hi,
> I updated petsc from 3.2 to 3.4. However, I found that the nonlinear solver previously worked well now failed with massage DIVERGED_LINE_SEARCH. 
> My problem is extremely nonlinear, cubic line search may fail but Newton method works.
>   
> With petsc 3.2, I only use basic Newton by calling 
> SNESLineSearchSet(snes, SNESLineSearchNo, PETSC_NULL) 
> With 3.4 I call basic Newton by  
> SNESLineSearchSetType(snes, SNESLINESEARCHBASIC).
> 
> However, petsc 3.4 still reports 
> [0] SNESSolve_NEWTONLS(): fnorm=3.7500000000000006e-02, gnorm=3.7500000000000006e-02, ynorm=1.0000000000000000e+00, lssucceed=0
> [0] SNESNEWTONLSCheckLocalMin_Private(): || J^T F|| 1.414213562373e+00 near zero implies found a local minimum  
> 
> 
> It seems petsc 3.4 can not disable line search. Does anyone have the same problem?
> 
> Gong Ding
>  
> 
> 

From rongliang.chan at gmail.com  Wed Oct 23 01:21:42 2013
From: rongliang.chan at gmail.com (Rongliang Chen)
Date: Wed, 23 Oct 2013 14:21:42 +0800
Subject: [petsc-users] How to debug the petsc no-debug version codes
Message-ID: <52676AF6.1010106@gmail.com>

Hi there,

My code woks well for the petsc-3.4.2 debug version, but get the 
following errors for the no-debug version. Do you have any suggestions 
for debugging the code?

Best,
Rongliang

-------------
[2]PETSC ERROR: [8]PETSC ERROR: 
------------------------------------------------------------------------
[8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[8]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[8]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[8]PETSC ERROR: to get more information on the crash.
[8]PETSC ERROR: --------------------- Error Message 
------------------------------------
[8]PETSC ERROR: Signal received!
[8]PETSC ERROR: 
------------------------------------------------------------------------
[8]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[8]PETSC ERROR: See docs/changes/index.html for recent updates.
[8]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[8]PETSC ERROR: See docs/index.html for manual pages.
[8]PETSC ERROR: 
------------------------------------------------------------------------
[8]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0804 by 
ronglian Tue Oct 22 22:06:05 2013
[8]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[8]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[8]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[8]PETSC ERROR: 
------------------------------------------------------------------------
[8]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
My Breakpoint: SetupASM Done !!!!
[186]PETSC ERROR: 
------------------------------------------------------------------------
[186]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[186]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[186]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[186]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[186]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[186]PETSC ERROR: to get more information on the crash.
[186]PETSC ERROR: --------------------- Error Message 
------------------------------------
[186]PETSC ERROR: [194]PETSC ERROR: [160]PETSC ERROR: [218]PETSC ERROR: 
[108]PETSC ERROR: 
------------------------------------------------------------------------
[108]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[108]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[108]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[108]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[108]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[108]PETSC ERROR: to get more information on the crash.
[108]PETSC ERROR: --------------------- Error Message 
------------------------------------
[108]PETSC ERROR: Signal received!
[108]PETSC ERROR: 
------------------------------------------------------------------------
[108]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[108]PETSC ERROR: See docs/changes/index.html for recent updates.
[108]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[90]PETSC ERROR: 
------------------------------------------------------------------------
[90]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[90]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[90]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[90]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[90]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[90]PETSC ERROR: to get more information on the crash.
[90]PETSC ERROR: --------------------- Error Message 
------------------------------------
[90]PETSC ERROR: Signal received!
[90]PETSC ERROR: 
------------------------------------------------------------------------
[90]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[90]PETSC ERROR: See docs/changes/index.html for recent updates.
[184]PETSC ERROR: [0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0804 by 
ronglian Tue Oct 22 22:06:05 2013
[0]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[0]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[0]PETSC ERROR: [24]PETSC ERROR: 
------------------------------------------------------------------------
[24]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[24]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[24]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[24]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[24]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[24]PETSC ERROR: to get more information on the crash.
[24]PETSC ERROR: --------------------- Error Message 
------------------------------------
[24]PETSC ERROR: Signal received!
[24]PETSC ERROR: 
------------------------------------------------------------------------
[24]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[24]PETSC ERROR: See docs/changes/index.html for recent updates.
[24]PETSC ERROR: [56]PETSC ERROR: 
------------------------------------------------------------------------
[56]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[56]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[56]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[56]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[56]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[56]PETSC ERROR: to get more information on the crash.
[76]PETSC ERROR: 
------------------------------------------------------------------------
[76]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[76]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[76]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[76]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[76]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[76]PETSC ERROR: to get more information on the crash.
[76]PETSC ERROR: --------------------- Error Message 
------------------------------------
[76]PETSC ERROR: Signal received!
[76]PETSC ERROR: 
------------------------------------------------------------------------
[76]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[76]PETSC ERROR: See docs/changes/index.html for recent updates.
[76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[168]PETSC ERROR: 
------------------------------------------------------------------------
[168]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[168]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[168]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[168]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[168]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[168]PETSC ERROR: to get more information on the crash.
[168]PETSC ERROR: --------------------- Error Message 
------------------------------------
[168]PETSC ERROR: Signal received!
[168]PETSC ERROR: 
------------------------------------------------------------------------
[168]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[168]PETSC ERROR: See docs/changes/index.html for recent updates.
[168]PETSC ERROR: [112]PETSC ERROR: [84]PETSC ERROR: Configure options 
--known-level1-dcache-size=32768 --known-level1-dcache-linesize=64 
--known-level1-dcache-assoc=8 --known-memcmp-ok=1 --known-sizeof-char=1 
--known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 
--known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[26]PETSC ERROR: [236]PETSC ERROR: 
------------------------------------------------------------------------
[236]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[236]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[236]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[236]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[236]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[236]PETSC ERROR: to get more information on the crash.
[236]PETSC ERROR: --------------------- Error Message 
------------------------------------
[236]PETSC ERROR: Signal received!
[236]PETSC ERROR: 
------------------------------------------------------------------------
[236]PETSC ERROR: [162]PETSC ERROR: 
------------------------------------------------------------------------
[162]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[162]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[162]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[162]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[162]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[162]PETSC ERROR: to get more information on the crash.
[162]PETSC ERROR: --------------------- Error Message 
------------------------------------
[162]PETSC ERROR: Signal received!
[162]PETSC ERROR: 
------------------------------------------------------------------------
[162]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[162]PETSC ERROR: See docs/changes/index.html for recent updates.
[162]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[162]PETSC ERROR: See docs/index.html for manual pages.
[162]PETSC ERROR: 
------------------------------------------------------------------------
[162]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0845 by 
ronglian Tue Oct 22 22:06:05 2013
[162]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[162]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[162]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[162]PETSC ERROR: 
------------------------------------------------------------------------
[162]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[64]PETSC ERROR: 
------------------------------------------------------------------------
[64]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[64]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[64]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[64]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[64]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[64]PETSC ERROR: to get more information on the crash.
[64]PETSC ERROR: --------------------- Error Message 
------------------------------------
[64]PETSC ERROR: Signal received!
[64]PETSC ERROR: 
------------------------------------------------------------------------
Signal received!
[186]PETSC ERROR: 
------------------------------------------------------------------------
[186]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[186]PETSC ERROR: See docs/changes/index.html for recent updates.
[186]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[186]PETSC ERROR: See docs/index.html for manual pages.
[186]PETSC ERROR: 
------------------------------------------------------------------------
[186]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0868 by 
ronglian Tue Oct 22 22:06:04 2013
[186]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[186]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[186]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[186]PETSC ERROR: 
------------------------------------------------------------------------
[186]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[12]PETSC ERROR: [192]PETSC ERROR: 
------------------------------------------------------------------------
[192]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[192]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[192]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[192]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[192]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[192]PETSC ERROR: to get more information on the crash.
[192]PETSC ERROR: --------------------- Error Message 
------------------------------------
[192]PETSC ERROR: Signal received!
[192]PETSC ERROR: 
------------------------------------------------------------------------
[192]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[192]PETSC ERROR: See docs/changes/index.html for recent updates.
[192]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[192]PETSC ERROR: See docs/index.html for manual pages.
[192]PETSC ERROR: 
------------------------------------------------------------------------
[192]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0870 by 
ronglian Tue Oct 22 22:06:05 2013
[192]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[192]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[192]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[192]PETSC ERROR: 
------------------------------------------------------------------------
[192]PETSC ERROR: [96]PETSC ERROR: [128]PETSC ERROR: [36]PETSC ERROR: 
[204]PETSC ERROR: 
------------------------------------------------------------------------
[204]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[204]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[204]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[204]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[204]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[204]PETSC ERROR: to get more information on the crash.
[204]PETSC ERROR: --------------------- Error Message 
------------------------------------
[204]PETSC ERROR: Signal received!
[204]PETSC ERROR: 
------------------------------------------------------------------------
[204]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[204]PETSC ERROR: See docs/changes/index.html for recent updates.
[204]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[204]PETSC ERROR: See docs/index.html for manual pages.
[204]PETSC ERROR: [208]PETSC ERROR: [114]PETSC ERROR: 
------------------------------------------------------------------------
[114]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[114]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[114]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[114]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[114]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[114]PETSC ERROR: to get more information on the crash.
[114]PETSC ERROR: --------------------- Error Message 
------------------------------------
[114]PETSC ERROR: Signal received!
[114]PETSC ERROR: 
------------------------------------------------------------------------
[114]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[114]PETSC ERROR: See docs/changes/index.html for recent updates.
[114]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[114]PETSC ERROR: See docs/index.html for manual pages.
[114]PETSC ERROR: 
------------------------------------------------------------------------
[114]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0818 by 
ronglian Tue Oct 22 22:06:05 2013
[114]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[114]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[114]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[114]PETSC ERROR: 
------------------------------------------------------------------------
[114]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[164]PETSC ERROR: 
------------------------------------------------------------------------
[164]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[164]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[164]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[164]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[164]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[164]PETSC ERROR: to get more information on the crash.
[164]PETSC ERROR: --------------------- Error Message 
------------------------------------
[164]PETSC ERROR: Signal received!
[164]PETSC ERROR: 
------------------------------------------------------------------------
[164]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[164]PETSC ERROR: See docs/changes/index.html for recent updates.
[164]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[164]PETSC ERROR: See docs/index.html for manual pages.
[164]PETSC ERROR: 
------------------------------------------------------------------------
[164]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0845 by 
ronglian Tue Oct 22 22:06:05 2013
[164]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[164]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[164]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[164]PETSC ERROR: 
------------------------------------------------------------------------
[164]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[144]PETSC ERROR: 
------------------------------------------------------------------------
[144]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[144]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[144]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[144]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[144]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[144]PETSC ERROR: to get more information on the crash.
[144]PETSC ERROR: --------------------- Error Message 
------------------------------------
[144]PETSC ERROR: Signal received!
[144]PETSC ERROR: 
------------------------------------------------------------------------
[144]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[144]PETSC ERROR: See docs/changes/index.html for recent updates.
[144]PETSC ERROR: [176]PETSC ERROR: 
------------------------------------------------------------------------
[176]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[176]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[176]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[176]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[176]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[176]PETSC ERROR: to get more information on the crash.
[176]PETSC ERROR: --------------------- Error Message 
------------------------------------
[176]PETSC ERROR: Signal received!
[176]PETSC ERROR: 
------------------------------------------------------------------------
[176]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[176]PETSC ERROR: See docs/changes/index.html for recent updates.
[176]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[176]PETSC ERROR: See docs/index.html for manual pages.
[176]PETSC ERROR: 
------------------------------------------------------------------------
[176]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0856 by 
ronglian Tue Oct 22 22:06:05 2013
[176]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[176]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[176]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[176]PETSC ERROR: 
------------------------------------------------------------------------
[176]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[82]PETSC ERROR: 
------------------------------------------------------------------------
[82]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[82]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[82]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[82]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[82]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[82]PETSC ERROR: to get more information on the crash.
[82]PETSC ERROR: --------------------- Error Message 
------------------------------------
[82]PETSC ERROR: Signal received!
[82]PETSC ERROR: 
------------------------------------------------------------------------
[82]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[82]PETSC ERROR: See docs/changes/index.html for recent updates.
[82]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[82]PETSC ERROR: See docs/index.html for manual pages.
[82]PETSC ERROR: 
------------------------------------------------------------------------
[82]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0811 by 
ronglian Tue Oct 22 22:06:05 2013
[82]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[82]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[82]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[82]PETSC ERROR: 
------------------------------------------------------------------------
[82]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[32]PETSC ERROR: 
------------------------------------------------------------------------
[32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[32]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[32]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[32]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[32]PETSC ERROR: to get more information on the crash.
[32]PETSC ERROR: --------------------- Error Message 
------------------------------------
[32]PETSC ERROR: Signal received!
[32]PETSC ERROR: 
------------------------------------------------------------------------
[32]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[32]PETSC ERROR: See docs/changes/index.html for recent updates.
[32]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[32]PETSC ERROR: See docs/index.html for manual pages.
[32]PETSC ERROR: 
------------------------------------------------------------------------
[32]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0737 by 
ronglian Tue Oct 22 22:06:04 2013
[32]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[32]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[32]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[32]PETSC ERROR: 
------------------------------------------------------------------------
[32]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[72]PETSC ERROR: [220]PETSC ERROR: 
------------------------------------------------------------------------
[220]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[220]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[220]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[220]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[220]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[220]PETSC ERROR: to get more information on the crash.
[220]PETSC ERROR: --------------------- Error Message 
------------------------------------
[220]PETSC ERROR: Signal received!
[220]PETSC ERROR: 
------------------------------------------------------------------------
[220]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[220]PETSC ERROR: See docs/changes/index.html for recent updates.
[220]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[220]PETSC ERROR: See docs/index.html for manual pages.
[220]PETSC ERROR: 
------------------------------------------------------------------------
[220]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0880 by 
ronglian Tue Oct 22 22:06:05 2013
[220]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[220]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[220]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[220]PETSC ERROR: 
------------------------------------------------------------------------
[220]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[178]PETSC ERROR: [68]PETSC ERROR: 
------------------------------------------------------------------------
[68]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[68]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[68]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[68]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[68]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[68]PETSC ERROR: to get more information on the crash.
[68]PETSC ERROR: --------------------- Error Message 
------------------------------------
[68]PETSC ERROR: Signal received!
[68]PETSC ERROR: 
------------------------------------------------------------------------
[68]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[68]PETSC ERROR: See docs/changes/index.html for recent updates.
[68]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[68]PETSC ERROR: See docs/index.html for manual pages.
[68]PETSC ERROR: 
------------------------------------------------------------------------
[68]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0810 by 
ronglian Tue Oct 22 22:06:05 2013
[68]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[68]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[68]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[68]PETSC ERROR: 
------------------------------------------------------------------------
[68]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[88]PETSC ERROR: 
------------------------------------------------------------------------
[88]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[88]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[88]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[88]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[88]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[88]PETSC ERROR: to get more information on the crash.
[88]PETSC ERROR: --------------------- Error Message 
------------------------------------
[88]PETSC ERROR: Signal received!
[88]PETSC ERROR: 
------------------------------------------------------------------------
[88]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[88]PETSC ERROR: See docs/changes/index.html for recent updates.
[88]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[88]PETSC ERROR: See docs/index.html for manual pages.
[88]PETSC ERROR: 
------------------------------------------------------------------------
[88]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0814 by 
ronglian Tue Oct 22 22:06:05 2013
[88]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[88]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[88]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[88]PETSC ERROR: 
------------------------------------------------------------------------
[88]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[196]PETSC ERROR: 
------------------------------------------------------------------------
[196]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[196]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[196]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[196]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[196]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[196]PETSC ERROR: to get more information on the crash.
[196]PETSC ERROR: --------------------- Error Message 
------------------------------------
[196]PETSC ERROR: Signal received!
[196]PETSC ERROR: 
------------------------------------------------------------------------
[196]PETSC ERROR: Petsc Release Version 3.4.2, unknown
[196]PETSC ERROR: See docs/changes/index.html for recent updates.
[196]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[196]PETSC ERROR: See docs/index.html for manual pages.
[196]PETSC ERROR: 
------------------------------------------------------------------------
[196]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0870 by 
ronglian Tue Oct 22 22:06:05 2013
[196]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-3.4.3/Janus-nodebug-64bit/lib
[196]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
[196]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit 
--download-exodusii=1 --download-hdf5=1 --download-netcdf=1 
--known-64-bit-blas-indices
[196]PETSC ERROR: 
------------------------------------------------------------------------
[196]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
[180]PETSC ERROR: 
------------------------------------------------------------------------
[180]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[180]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[180]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[180]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[180]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[180]PETSC ERROR: to get more information on the crash.
[180]PETSC ERROR: --------------------- Error Message 
------------------------------------
[180]PETSC ERROR: Signal received!

From juhaj at iki.fi  Wed Oct 23 02:57:03 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Wed, 23 Oct 2013 09:57:03 +0200
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <8738oewbv9.fsf@mcs.anl.gov>
References: <2569883.TWAWmgZMUo@rigel> <7639338.FUb38SSzMH@rigel>
	<8738oewbv9.fsf@mcs.anl.gov>
Message-ID: <5862566.rSSLWvQenN@rigel>

Hi Jed, 

My first attempt at sanitising chunking.

The rationale is to keep the old behaviour if possible, so I just add a if-
else after the chunk sizes are first calculated and change those sizes that 
need to be changed. [There is also a zeroing of the 6-element arrays in the 
beginning of the function because I multiply them together to find the various 
sizes and at least Cray seems to have random numbers in the stack when 
entering the function, causing the sizes to be crazy.]

I first check if the average vector size exceeds 4 GiB and if it does, simply 
use as few chunks as possible, splitting along slowest varying axis first and 
if that is not enough, splitting the next axis, too etc. The patch *can* split 
the fastest varying dimension, too, but that seems overkill: it would mean 
that da_grid_x*dof > 4 GiB, which seems unlikely. It cannot split dof.

If the average Vec size is <= 4 GiB, but total Vec size is > 4 GiB, there is a 
simple logic: first see if <local_z_size>*<global_y_size>*<global_x_size>*dof 
is smaller than "target size". Target size comes from your formula earlier in 
this thread. If that is so, use that, if not, then just use the local Vec size 
as the chunk size. This could be improved by having an intermediate size 
solution, too, but this solution gives *my* tests decent performance across 
several global sizes, so it does not seem to be too bad. Now, the performance 
I get is just ~3 GiB/s, using 36 OSTs on a lustre. Using one OST I get ~200 
MiB/s, so the scaling is not that bad: I get half of 36*200 MiB/s, so I think 
my performance is limited by the OSTs, not MPI-IO or HDF5 or chunking. Tests 
with bigger systems would of course be needed to determine whether this 
solution works for truly huge files, too: the biggest I tried are just over 50 
GiB. Because I'm forced to spend my research quota on this, I don't want to 
waste too much of it by going to ridiculous size files (these are about the 
biggest ones I plan to use in practice for now).

And the patch itself is attached, of course. Please improve if you think it 
should be improved and adapt to PETSc coding conventions if necessary (at 
least the #defines are in a funny place to keep the patch simple).

Cheers,
Juha

On Sunday 06 Oct 2013 12:24:58 Jed Brown wrote:
> Juha J?ykk? <juhaj at iki.fi> writes:
> > Argh, messy indeed. Are you sure you mean 65 k and not 64 Ki?
> 
> I was using 65 as shorthand for 2^{32}, yes.
> 
> > I made a small table of the situation just to make sure I am not
> > missing anything. In the table, "small" means < 4 GB, "large" means >=
> > 4 GB, "few" means < 65 k, "many" means >= 65 k. Note that local size >
> > global size is impossible, but I include the row on the table for
> > completeness's sake.
> > 
> > Variables: 	local size	global size	# ranks		chunks
> > 
> > 		small		small		few		global size
> > 		small		small		many		global size[1]
> > 		small		large		few		avg local size
> > 		small		large		many		4 GiB
> > 		large		small		few		impossible
> > 		large		small		many		impossible
> > 		large		large		few		4 GiB[2]
> > 		large		large		many		65 k chunks
> 
> The last line cannot be stored since it consists of more than 65k chunks
> of size larger than 4 GiB.
> 
> > [1] It sounds improbable anyone would run a problem with < 4 GiB data with
> > >= 65k ranks, but fortunately it's not a problem.
> 
> Au contraire, it's typical for strong scaling to roll over at about 1000
> dofs/core (about 10 kB of data), so 65k ranks is still less than 1 GB.
> 
> > [2] Unless I'm mistaken, this situation will always give < 65 k chunks for
> > 4 GiB chunk size.
> 
> Well, 33k ranks each trying to write 8 GiB would need 66k chunks, but
> there is no way to write this file so we don't need to worry about it.
> 
> > I also believe your formula gives "the right" answer in each case. Just
> > one
> > more question: is "average local size" a good solution or is it better to
> > use "max local size"? The latter will cause more unnecessary data in the
> > file, but unless I'm mistaken, the former will require extra MPI
> > communication to fill in the portions of ranks whose local size is less
> > than average.
> 
> It depends how the compute nodes are connected to the file system, but
> even if you use "average" and if size is statistically uncorrelated with
> rank but has positive variance, the expected value of the skew will be
> more than a rank's contribution for sufficiently large numbers of cores.
> In other words, it's not possible to "balance" without needing to move
> all data for a sufficiently large number of ranks.
> 
> > HDF5 really needs to fix this internally. As it stands, a single HDF5
> > dataset cannot hold more than 260 TiB ? not that many people would want
> > such files anyway, but then again, "640 kiB should be enough for
> > everybody", right? I'm running simulations which take more than terabyte
> > of memory, and I'm by far not the biggest memory consumer in the world,
> > so the limit is not really as far as it might seem.
> 
> We're already there.  Today's large machines can fit several vectors of
> size 260 TB in memory.
> 
> >> I think we're planning to tag 3.4.3 in the next couple weeks.  There
> >> might be a 3.4.4 as well, but I could see going straight to 3.5.
> > 
> > Ok. I don't see myself having time to fix and test this in two weeks, but
> > 3.4.4 should be doable. Anyone else want to fix the bug by then?
> 
> I'll write back if I get to it.
-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hdf5_chunks.patch
Type: text/x-patch
Size: 5397 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/b0e769d8/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/b0e769d8/attachment.pgp>

From knepley at gmail.com  Wed Oct 23 02:57:47 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 23 Oct 2013 02:57:47 -0500
Subject: [petsc-users] How to debug the petsc no-debug version codes
In-Reply-To: <52676AF6.1010106@gmail.com>
References: <52676AF6.1010106@gmail.com>
Message-ID: <CAMYG4G=AGHO27QJQYsQcMVT3OYU5CMhoEp_YZXBwHANonMhqkA@mail.gmail.com>

On Wed, Oct 23, 2013 at 1:21 AM, Rongliang Chen <rongliang.chan at gmail.com>wrote:

> Hi there,
>
> My code woks well for the petsc-3.4.2 debug version, but get the following
> errors for the no-debug version. Do you have any suggestions for debugging
> the code?
>

The stack trace shows that the SEGV happens in non-PETSc code:

  [8]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file

Since it does not happen in the debugging version, it is likely that an
uninitialized variable is
being used to index into memory. Make sure you turn on all compiler
warnings and pay attention
to them in your code. Also, run it through valgrind.

   Matt


> Best,
> Rongliang
>
> -------------
> [2]PETSC ERROR: [8]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [8]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[8]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[8]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [8]PETSC ERROR: to get more information on the crash.
> [8]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [8]PETSC ERROR: Signal received!
> [8]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [8]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [8]PETSC ERROR: See docs/changes/index.html for recent updates.
> [8]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [8]PETSC ERROR: See docs/index.html for manual pages.
> [8]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [8]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0804 by
> ronglian Tue Oct 22 22:06:05 2013
> [8]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [8]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [8]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [8]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [8]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> My Breakpoint: SetupASM Done !!!!
> [186]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [186]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [186]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [186]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[186]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[186]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [186]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [186]PETSC ERROR: to get more information on the crash.
> [186]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [186]PETSC ERROR: [194]PETSC ERROR: [160]PETSC ERROR: [218]PETSC ERROR:
> [108]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [108]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [108]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [108]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[108]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[108]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [108]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [108]PETSC ERROR: to get more information on the crash.
> [108]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [108]PETSC ERROR: Signal received!
> [108]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [108]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [108]PETSC ERROR: See docs/changes/index.html for recent updates.
> [108]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [90]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [90]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [90]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [90]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[90]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[90]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [90]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [90]PETSC ERROR: to get more information on the crash.
> [90]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [90]PETSC ERROR: Signal received!
> [90]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [90]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [90]PETSC ERROR: See docs/changes/index.html for recent updates.
> [184]PETSC ERROR: [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[0]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0804 by
> ronglian Tue Oct 22 22:06:05 2013
> [0]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [0]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [0]PETSC ERROR: [24]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [24]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [24]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [24]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[24]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[24]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [24]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [24]PETSC ERROR: to get more information on the crash.
> [24]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [24]PETSC ERROR: Signal received!
> [24]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [24]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [24]PETSC ERROR: See docs/changes/index.html for recent updates.
> [24]PETSC ERROR: [56]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [56]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [56]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [56]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[56]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[56]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [56]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [56]PETSC ERROR: to get more information on the crash.
> [76]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [76]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [76]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [76]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[76]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[76]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [76]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [76]PETSC ERROR: to get more information on the crash.
> [76]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [76]PETSC ERROR: Signal received!
> [76]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [76]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [76]PETSC ERROR: See docs/changes/index.html for recent updates.
> [76]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [168]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [168]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [168]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [168]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[168]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[168]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [168]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [168]PETSC ERROR: to get more information on the crash.
> [168]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [168]PETSC ERROR: Signal received!
> [168]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [168]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [168]PETSC ERROR: See docs/changes/index.html for recent updates.
> [168]PETSC ERROR: [112]PETSC ERROR: [84]PETSC ERROR: Configure options
> --known-level1-dcache-size=**32768 --known-level1-dcache-**linesize=64
> --known-level1-dcache-assoc=8 --known-memcmp-ok=1 --known-sizeof-char=1
> --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4
> --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4
> --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8
> --known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4
> --known-mpi-long-double=1 --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [26]PETSC ERROR: [236]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [236]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [236]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [236]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[236]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[236]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [236]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [236]PETSC ERROR: to get more information on the crash.
> [236]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [236]PETSC ERROR: Signal received!
> [236]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [236]PETSC ERROR: [162]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [162]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [162]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [162]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[162]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[162]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [162]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [162]PETSC ERROR: to get more information on the crash.
> [162]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [162]PETSC ERROR: Signal received!
> [162]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [162]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [162]PETSC ERROR: See docs/changes/index.html for recent updates.
> [162]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [162]PETSC ERROR: See docs/index.html for manual pages.
> [162]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [162]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0845 by
> ronglian Tue Oct 22 22:06:05 2013
> [162]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [162]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [162]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [162]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [162]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [64]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [64]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [64]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [64]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[64]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[64]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [64]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [64]PETSC ERROR: to get more information on the crash.
> [64]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [64]PETSC ERROR: Signal received!
> [64]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> Signal received!
> [186]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [186]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [186]PETSC ERROR: See docs/changes/index.html for recent updates.
> [186]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [186]PETSC ERROR: See docs/index.html for manual pages.
> [186]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [186]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0868 by
> ronglian Tue Oct 22 22:06:04 2013
> [186]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [186]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [186]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [186]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [186]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [12]PETSC ERROR: [192]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [192]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [192]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [192]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[192]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[192]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [192]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [192]PETSC ERROR: to get more information on the crash.
> [192]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [192]PETSC ERROR: Signal received!
> [192]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [192]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [192]PETSC ERROR: See docs/changes/index.html for recent updates.
> [192]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [192]PETSC ERROR: See docs/index.html for manual pages.
> [192]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [192]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0870 by
> ronglian Tue Oct 22 22:06:05 2013
> [192]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [192]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [192]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [192]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [192]PETSC ERROR: [96]PETSC ERROR: [128]PETSC ERROR: [36]PETSC ERROR:
> [204]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [204]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [204]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [204]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[204]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[204]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [204]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [204]PETSC ERROR: to get more information on the crash.
> [204]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [204]PETSC ERROR: Signal received!
> [204]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [204]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [204]PETSC ERROR: See docs/changes/index.html for recent updates.
> [204]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [204]PETSC ERROR: See docs/index.html for manual pages.
> [204]PETSC ERROR: [208]PETSC ERROR: [114]PETSC ERROR:
> ------------------------------**------------------------------**
> ------------
> [114]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [114]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [114]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[114]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[114]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [114]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [114]PETSC ERROR: to get more information on the crash.
> [114]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [114]PETSC ERROR: Signal received!
> [114]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [114]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [114]PETSC ERROR: See docs/changes/index.html for recent updates.
> [114]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [114]PETSC ERROR: See docs/index.html for manual pages.
> [114]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [114]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0818 by
> ronglian Tue Oct 22 22:06:05 2013
> [114]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [114]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [114]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [114]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [114]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [164]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [164]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [164]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [164]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[164]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[164]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [164]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [164]PETSC ERROR: to get more information on the crash.
> [164]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [164]PETSC ERROR: Signal received!
> [164]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [164]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [164]PETSC ERROR: See docs/changes/index.html for recent updates.
> [164]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [164]PETSC ERROR: See docs/index.html for manual pages.
> [164]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [164]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0845 by
> ronglian Tue Oct 22 22:06:05 2013
> [164]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [164]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [164]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [164]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [164]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [144]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [144]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [144]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [144]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[144]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[144]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [144]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [144]PETSC ERROR: to get more information on the crash.
> [144]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [144]PETSC ERROR: Signal received!
> [144]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [144]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [144]PETSC ERROR: See docs/changes/index.html for recent updates.
> [144]PETSC ERROR: [176]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [176]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [176]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [176]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[176]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[176]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [176]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [176]PETSC ERROR: to get more information on the crash.
> [176]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [176]PETSC ERROR: Signal received!
> [176]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [176]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [176]PETSC ERROR: See docs/changes/index.html for recent updates.
> [176]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [176]PETSC ERROR: See docs/index.html for manual pages.
> [176]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [176]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0856 by
> ronglian Tue Oct 22 22:06:05 2013
> [176]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [176]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [176]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [176]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [176]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [82]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [82]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [82]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [82]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[82]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[82]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [82]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [82]PETSC ERROR: to get more information on the crash.
> [82]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [82]PETSC ERROR: Signal received!
> [82]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [82]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [82]PETSC ERROR: See docs/changes/index.html for recent updates.
> [82]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [82]PETSC ERROR: See docs/index.html for manual pages.
> [82]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [82]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0811 by
> ronglian Tue Oct 22 22:06:05 2013
> [82]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [82]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [82]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [82]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [82]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [32]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [32]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[32]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[32]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [32]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [32]PETSC ERROR: to get more information on the crash.
> [32]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [32]PETSC ERROR: Signal received!
> [32]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [32]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [32]PETSC ERROR: See docs/changes/index.html for recent updates.
> [32]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [32]PETSC ERROR: See docs/index.html for manual pages.
> [32]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [32]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0737 by
> ronglian Tue Oct 22 22:06:04 2013
> [32]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [32]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [32]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [32]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [32]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [72]PETSC ERROR: [220]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [220]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [220]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [220]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[220]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[220]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [220]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [220]PETSC ERROR: to get more information on the crash.
> [220]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [220]PETSC ERROR: Signal received!
> [220]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [220]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [220]PETSC ERROR: See docs/changes/index.html for recent updates.
> [220]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [220]PETSC ERROR: See docs/index.html for manual pages.
> [220]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [220]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0880 by
> ronglian Tue Oct 22 22:06:05 2013
> [220]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [220]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [220]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [220]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [220]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [178]PETSC ERROR: [68]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [68]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [68]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [68]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[68]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[68]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [68]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [68]PETSC ERROR: to get more information on the crash.
> [68]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [68]PETSC ERROR: Signal received!
> [68]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [68]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [68]PETSC ERROR: See docs/changes/index.html for recent updates.
> [68]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [68]PETSC ERROR: See docs/index.html for manual pages.
> [68]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [68]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0810 by
> ronglian Tue Oct 22 22:06:05 2013
> [68]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [68]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [68]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [68]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [68]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [88]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [88]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [88]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [88]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[88]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[88]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [88]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [88]PETSC ERROR: to get more information on the crash.
> [88]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [88]PETSC ERROR: Signal received!
> [88]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [88]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [88]PETSC ERROR: See docs/changes/index.html for recent updates.
> [88]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [88]PETSC ERROR: See docs/index.html for manual pages.
> [88]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [88]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0814 by
> ronglian Tue Oct 22 22:06:05 2013
> [88]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [88]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [88]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [88]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [88]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [196]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [196]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [196]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [196]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[196]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[196]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [196]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [196]PETSC ERROR: to get more information on the crash.
> [196]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [196]PETSC ERROR: Signal received!
> [196]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [196]PETSC ERROR: Petsc Release Version 3.4.2, unknown
> [196]PETSC ERROR: See docs/changes/index.html for recent updates.
> [196]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [196]PETSC ERROR: See docs/index.html for manual pages.
> [196]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [196]PETSC ERROR: ./fsi3d on a Janus-nodebug-64bit named node0870 by
> ronglian Tue Oct 22 22:06:05 2013
> [196]PETSC ERROR: Libraries linked from /projects/ronglian/soft/petsc-**
> 3.4.3/Janus-nodebug-64bit/lib
> [196]PETSC ERROR: Configure run at Thu Oct 17 01:29:46 2013
> [196]PETSC ERROR: Configure options --known-level1-dcache-size=**32768
> --known-level1-dcache-**linesize=64 --known-level1-dcache-assoc=8
> --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8
> --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8
> --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8
> --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=8
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1
> --known-mpi-c-double-complex=0 --download-blacs=1
> --download-f-blas-lapack=1 --download-metis=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu_dist=1
> --known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1
> --with-debugging=0 --with-mpi-shared=1 PETSC_ARCH=Janus-nodebug-64bit
> --download-exodusii=1 --download-hdf5=1 --download-netcdf=1
> --known-64-bit-blas-indices
> [196]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [196]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [180]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [180]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [180]PETSC ERROR: Try option -start_in_debugger or
> -on_error_attach_debugger
> [180]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> documentation/faq.html#**valgrind[180]PETSC<http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[180]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [180]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [180]PETSC ERROR: to get more information on the crash.
> [180]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [180]PETSC ERROR: Signal received!
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/23f3be9a/attachment-0001.html>

From juhaj at iki.fi  Wed Oct 23 03:10:27 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Wed, 23 Oct 2013 10:10:27 +0200
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <1677196.9ZmfKVC8up@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
	<1677196.9ZmfKVC8up@dhcp071>
Message-ID: <1665283.VQd2nzgDvj@rigel>

> I will test with 3.3 later today if I have the time, but I'm pretty sure
> things were "better" there.

I did happen to have one system still having 3.2 on it, so I tested that 
instead of 3.3 and at least 3.2 seems to be a bit more conservative in 
numbers. For the same $CMD as before, I get

for x in 1 2 4 ; do mpirun -np $x python -c "$CMD"; done
21 MiB / 21868 kB
21 MiB / 21888 kB
218 MiB / 224028 kB
22 MiB / 22852 kB
22 MiB / 22600 kB
22 MiB / 22620 kB
22 MiB / 22872 kB
79 MiB / 81640 kB
79 MiB / 81372 kB
22 MiB / 22628 kB
22 MiB / 22648 kB
22 MiB / 22632 kB
22 MiB / 22652 kB
22 MiB / 22880 kB
22 MiB / 22900 kB
22 MiB / 22908 kB
22 MiB / 22928 kB
51 MiB / 52944 kB
51 MiB / 52936 kB
51 MiB / 53188 kB
51 MiB / 53216 kB

For comparison, 3.4.2 gives (from the previous email): 354 MiB for 1 rank, 141 
for 2 ranks and 81 for 4 ranks, which is a LOT less. I suspect this might have 
something to do with the DA -> DMDA change?

I still haven't got around testig Barry's branch, but hope to be able to do 
that over later this week.

Cheers,
Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/fa29af1e/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 23 06:20:10 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 06:20:10 -0500
Subject: [petsc-users] How to debug the petsc no-debug version codes
In-Reply-To: <52676AF6.1010106@gmail.com>
References: <52676AF6.1010106@gmail.com>
Message-ID: <871u3ckzdh.fsf@mcs.anl.gov>

Rongliang Chen <rongliang.chan at gmail.com> writes:
> My code woks well for the petsc-3.4.2 debug version, but get the 
> following errors for the no-debug version. Do you have any suggestions 
> for debugging the code?

Reducing the number of processes to reproduce the error always helps for
debugging.

One of the most common reasons is that a stack variable is initialized
to 0 at -O0, but contains garbage at higher optimization levels.  To
debug, use a debugging PETSc, but compile with "-O1 -g" and make sure
all warnings are turned on (at least -Wall, preferably -Wextra).
Make sure the code is valgrind-clean in this mode.

Another possibility is that debug-mode memory allocation/macros are
causing the different behavior.  First try turning off the tracing
malloc (run program with -malloc 0).  Then try an optimized PETSc built
with debugging (--with-debugging=0 COPTFLAGS='-O0 -g', or
--with-debugging=0 COPTFLAGS='-O1 -g').
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/09272ae4/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 23 07:39:19 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 07:39:19 -0500
Subject: [petsc-users] Unable to create >4GB sized HDF5 files on Cray
	XC30
In-Reply-To: <5862566.rSSLWvQenN@rigel>
References: <2569883.TWAWmgZMUo@rigel> <7639338.FUb38SSzMH@rigel>
	<8738oewbv9.fsf@mcs.anl.gov> <5862566.rSSLWvQenN@rigel>
Message-ID: <87li1kjh54.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:

> Hi Jed, 
>
> My first attempt at sanitising chunking.

Thanks, the logic looks reasonable.  Unfortunately, I don't think we can
have comprehensive tests because the large file sizes won't work in many
test environments.  That said, we could have a configurable parameter
instead of hard-coding 4 GiB, in which case we could set it to 1 MiB (or
less) for testing.

Could you make a commit locally and use "git format-patch" to prepare a
patch (or use a bitbucket pull request)?  The body of your email would
make a fantastic commit message.  Instructions here:

https://bitbucket.org/petsc/petsc/wiki/Home

Further comments below.

> --- gr2.c.orig	2013-05-14 03:17:51.000000000 +0300
> +++ gr2.c	2013-10-21 19:14:11.000000000 +0300
> @@ -341,6 +341,15 @@
>    const char     *vecname;
>    PetscErrorCode ierr;
>  
> +  // just in case some libraries do not empty memory for local variables:

We have to use C89-style comments because of Microsoft.

In the C language, variables are not automatically initialized.  You can
do it in the declaration using

  hsize_t maxDims[6] = {0},dims[6] = {0}, ...

> +  for (dim=0;dim<6;dim++) {
> +    chunkDims[dim] = 0;
> +    maxDims[dim] = 0;
> +    dims[dim] = 0;
> +    count[dim] = 0;
> +    offset[dim] = 0;
> +  }
> +
>    PetscFunctionBegin;

We cannot have executable statements before PetscFunctionBegin.

>    ierr = PetscViewerHDF5OpenGroup(viewer, &file_id, &group);CHKERRQ(ierr);
>    ierr = PetscViewerHDF5GetTimestep(viewer, &timestep);CHKERRQ(ierr);
> @@ -395,6 +405,103 @@
>    chunkDims[dim] = dims[dim];
>    ++dim;
>  #endif
> +
> +  // some requirements for the chunks (assuming no split along time-step)
> +  int ranks;

Declarations cannot be mixed with statements in C89.

> +  MPI_Comm_size(PETSC_COMM_WORLD, &ranks);

PETSC_COMM_WORLD cannot be used in library code because this has to be
work when the vector resides on a subcommunicator.  Use
PetscObjectComm((PetscObject)xin).  And please name the variable "size"
or "comm_size".

> +  hsize_t chunk_size, target_size, dim_save,
> +    vec_size = sizeof(double)*da->M*da->N*da->P*da->w,
> +    avg_local_vec_size = (hsize_t) ceil(vec_size*1.0/ranks),
> +    max_chunks = 65536,
> +    max_chunk_size = 4294967296,
> +    GiB = 1073741824,

There is a potential problem with these long integer literals, I believe
with some C++ compilers prior to C++11 since they are not guaranteed to
interpret the large integer literal as having sufficiently large type.
One reasonable thing is to write

  hsize_t Ki = 1024;

and then

  Gi = Ki*Ki*Ki;


I would prefer to move all the logic below into a function
VecGetHDF5ChunkSize().  Note that almost the same logic appears in
VecView_MPI_HDF5 (src/vec/vec/impls/mpi/pdvec.c) so both versions should
call this function.  VecView_MPI_HDF5 can pass in a 1D (or 2D due to
blocks) global dimensions.

> +#define fmin3(a,b,c)	fmin(a,fmin(b,c))
> +#define fmax3(a,b,c)	fmax(a,fmax(b,c))
> +#define max(a,b)        (a>=b?a:b)

Please use PetscMax() and PetscMin().  You can define the 3-way versions
at the top of the file if need be.

> +  dim_save=dim;
> +  target_size = (hsize_t) ceil(fmin3(vec_size, fmax3(avg_local_vec_size, ceil(vec_size*1.0/max_chunks), min_size), max_chunk_size));
> +  chunk_size = (hsize_t) max(1,chunkDims[0])*max(1,chunkDims[1])*max(1,chunkDims[2])*max(1,chunkDims[3])*max(1,chunkDims[4])*max(1,chunkDims[5])*sizeof(PetscScalar);
> +
> +  // if size/rank > max_chunk_size, we need radical measures: even going down to
> +  // avg_local_vec_size is not enough, so we simply use chunk size of 4 GiB no matter
> +  // what, composed in the most efficient way possible.
> +  // N.B. this minimises the number of chunks, which may or may not be the optimal
> +  // solution. In a BG, for example, the optimal solution is probably to make # chunks = #
> +  // IO nodes involved, but this author has no access to a BG to figure out how to
> +  // reliably find the right number. And even then it may or may not be enough.
> +  if (avg_local_vec_size > max_chunk_size) {
> +    // check if we can just split local z-axis: is that enough?
> +    zslices=(hsize_t) ceil(vec_size*1.0/(da->p*max_chunk_size))*zslices;
> +    if (zslices > da->P) {
> +      // lattice is too large in xy-directions, splitting z only is not enough
> +      zslices = da->P;
> +      yslices=(hsize_t) ceil(vec_size*1.0/(zslices*da->n*max_chunk_size))*yslices;
> +      if (yslices > da->N) {
> +	// lattice is too large in x-direction, splitting along z, y is not enough
> +	yslices = da->N;

You mixed tabs and spaces here.  Please use all spaces.

> +	xslices=(hsize_t) ceil(vec_size*1.0/(zslices*yslices*da->m*max_chunk_size))*xslices;
> +      }
> +    }
> +    dim = 0;
> +    if (timestep >= 0) {
> +      ++dim;
> +    }
> +    // prefer to split z-axis, even down to planar slices
> +    if (da->dim == 3) {
> +      chunkDims[dim++] = (hsize_t) floor(da->P*1.0/zslices);
> +      chunkDims[dim++] = (hsize_t) floor(da->N*1.0/yslices);
> +      chunkDims[dim++] = (hsize_t) floor(da->M*1.0/xslices);
> +    } else {
> +      // This is a 2D world exceeding 4GiB in size; yes, I've seen them, even used myself
> +      chunkDims[dim++] = (hsize_t) floor(da->N*1.0/yslices);
> +      chunkDims[dim++] = (hsize_t) floor(da->M*1.0/xslices);
> +    }
> +    chunk_size = (hsize_t) fmax(1,chunkDims[0])*fmax(1,chunkDims[1])*fmax(1,chunkDims[2])*fmax(1,chunkDims[3])*fmax(1,chunkDims[4])*fmax(1,chunkDims[5])*sizeof(double);
> +  } else {
> +    if (target_size < chunk_size) {
> +      // only change the defaults if target_size < chunk_size
> +      dim = 0;
> +      if (timestep >= 0) {
> +	++dim;
> +      }
> +      // prefer to split z-axis, even down to planar slices
> +      if (da->dim == 3) {
> +	// try splitting the z-axis to core-size bits, i.e. divide chunk size by # ranks in z-direction
> +	if (target_size >= chunk_size/da->p) {
> +	  // just make chunks the size of <local_z>x<whole_world_y>x<whole_world_x>x<dof>
> +	  chunkDims[dim] = (hsize_t) ceil(da->P*1.0/da->p);
> +	} else {
> +	  // oops, just splitting the z-axis is NOT ENOUGH, need to split more; let's be
> +	  // radical and let everyone write all they've got
> +	  chunkDims[dim++] = (hsize_t) ceil(da->P*1.0/da->p);
> +	  chunkDims[dim++] = (hsize_t) ceil(da->N*1.0/da->n);
> +	  chunkDims[dim++] = (hsize_t) ceil(da->M*1.0/da->m);
> +	}
> +      } else {
> +	// This is a 2D world exceeding 4GiB in size; yes, I've seen them, even used myself
> +	if (target_size >= chunk_size/da->n) {
> +	  // just make chunks the size of <local_z>x<whole_world_y>x<whole_world_x>x<dof>
> +	  chunkDims[dim] = (hsize_t) ceil(da->N*1.0/da->n);
> +	} else {
> +	  // oops, just splitting the z-axis is NOT ENOUGH, need to split more; let's be
> +	  // radical and let everyone write all they've got
> +	  chunkDims[dim++] = (hsize_t) ceil(da->N*1.0/da->n);
> +	  chunkDims[dim++] = (hsize_t) ceil(da->M*1.0/da->m);
> +	}
> +      
> +      }
> +      chunk_size = (hsize_t) fmax(1,chunkDims[0])*fmax(1,chunkDims[1])*fmax(1,chunkDims[2])*fmax(1,chunkDims[3])*fmax(1,chunkDims[4])*fmax(1,chunkDims[5])*sizeof(double);
> +    } else {
> +      // precomputed chunks are fine, we don't need to do anything
> +    }
> +  }
> +  dim=dim_save; // need to have the original value here no matter what!
> +
>    filespace = H5Screate_simple(dim, dims, maxDims);
>    if (filespace == -1) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"Cannot H5Screate_simple()");
>  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/60a412ef/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 23 08:41:40 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 08:41:40 -0500
Subject: [petsc-users] VISS and VIRS convergence problems
In-Reply-To: <57596E6532C9F54F9DF09BCDFE7B1D3A86466166@exmbx06-cdc.nexus.csiro.au>
References: <57596E6532C9F54F9DF09BCDFE7B1D3A86466166@exmbx06-cdc.nexus.csiro.au>
Message-ID: <87k3h4je97.fsf@mcs.anl.gov>

Andrew, sorry about the silence.

Andrew.Wilkins at csiro.au writes:

> I've got a problem where I'm using viss or virs (petsc3.3) to enforce
> a lower bound on my solution.  The actual application is unsaturated
> Darcy-Richards single-phase flow - slow water flow through a porous
> medium.  As water is withdrawn from a region, the pressure drops, but
> physically the water pressure is bounded from below.  The DE ensures
> this via a nonlinear relative permeability function which goes to zero
> at the lower-bound of pressure: permeability=0 means no water can flow
> from the region.  However, in the linearization, the nonlinear solver
> won't know about this function going to zero (unless it just happens
> to end up right at the lower bound), so often the pressure goes into
> the unphysical region (pressure < lower) during the solution process.

Is the spatial discretization nominally monotone and you're only seeing
this due to attempt to take large time steps, or is there no discrete
positivity preservation?

> Therefore I'm using the VI solvers.
>
> First question - the -snes_vi_type appears to make no difference at all.  What's it supposed to do?

Some parts of SNESVI* are in disarray and they are certainly not up to
the standard of the rest of PETSc.  This documentation problem with
-snes_vi_type showed up in the refactoring commit below (between v3.2
and v3.3).  We now use the SNES types SNESVISS and SNESVIRS (actually,
SNESVINEWTONSSLS and SNESVINEWTONRSLS in v3.4).

The RS stands for "reduced space"; these methods eliminate those
variables that are in contact while satisfying the inequality and
perform Newton iterations in the remaining subspace.  The SS stands for
semi-smooth.  They solve modified problems in the full space.  SS
methods have some nicer nonlinear convergence properties, but the linear
systems are notoriously ill-conditioned.

commit c2fc9fa91a764ea1bc445c5db6fc55248852cdef
Author: Barry Smith <bsmith at mcs.anl.gov>
Date:   Sat Nov 5 20:30:54 2011 -0500

    reorganized SNESVI to two solvers SNESVIRS and SNESVISS  in seperate directories without being intertwined though they share some common code
    removed rsaug from being built since is broken
    
    Hg-commit: 31c8ce4101ddfbfe4fc6d005d66ef14f58ee082a

 include/petscsnes.h                          |    5 +-
 include/private/snesimpl.h                   |   14 +
 include/private/vecimpl.h                    |    1 +
 src/docs/website/documentation/faq.html      |    2 +-
 src/mat/impls/sbaij/mpi/mpisbaij.c           |   24 +-
 src/snes/examples/tutorials/ex54.c           |    3 +-
 src/snes/examples/tutorials/ex55.c           |    3 +-
 src/snes/examples/tutorials/ex60.c           |    5 +-
 src/snes/examples/tutorials/ex61.c           |    6 +-
 src/snes/examples/tutorials/ex63.c           |    5 +-
 src/snes/examples/tutorials/ex633D_DB.c      |    3 +-
 src/snes/examples/tutorials/ex64.c           |    5 +-
 src/snes/examples/tutorials/ex65.c           |    5 +-
 src/snes/examples/tutorials/ex653D.c         |    7 +-
 src/snes/examples/tutorials/makefile         |    2 +-
 src/snes/impls/vi/makefile                   |    3 +-
 src/snes/impls/vi/{ => rs}/makefile          |    8 +-
 src/snes/impls/vi/rs/virs.c                  |  795 ++++++++
 src/snes/impls/vi/rs/virsimpl.h              |   18 +
 src/snes/impls/vi/{ => rsaug}/makefile       |    8 +-
 src/snes/impls/vi/{vi.c => rsaug/virsaug.c}  |    0
 .../vi/{viimpl.h => rsaug/virsaugimpl.h}     |    2 +-
 src/snes/impls/vi/{ => ss}/makefile          |    8 +-
 src/snes/impls/vi/ss/viss.c                  |  499 +++++
 .../impls/vi/{viimpl.h => ss/vissimpl.h}     |   18 +-
 src/snes/impls/vi/vi.c                       | 1687 +---------------
 src/snes/interface/snesregi.c                |    6 +-
 src/sys/examples/tests/makefile              |    5 +-
 .../tutorials/phasefield/biharmonic.c        |   33 +-
 src/ts/examples/tutorials/phasefield/heat.c  |   52 +-
 30 files changed, 1499 insertions(+), 1733 deletions(-)


> Second question - So far, viss has converged always, but virs
> sometimes gives problems.  I've read in one email-thread that virs is
> preferred.  Why is that?  

The linear systems tend to be easier to solve.  RS methods are not
immune to "chatter".

> I'm basing my comment on only about 10 tests, but I'm still interested
> in your replies.  I'm pretty sure my jacobian is correct, as I have
> tested it using -snes_type=test, and also I have run using
> finite-differences, and still got the behaviour below.  This is the
> type of behaviour I get:
>
> DT: 2.885464e+03
>  Solving time step 48, time=406142.00...
>   Initial |residual|_2 = 1.536061e-01
>   NL step  0, |residual|_2 = 1.536061e-01
>   0 SNES VI Function norm 1.536061428302e-01 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
>     0 KSP Residual norm 1.536061428302e-01
>     1 KSP Residual norm 5.954022644970e-05
>     2 KSP Residual norm 1.874684877465e-07
>   NL step  1, |residual|_2 = 2.508324e-02
>   1 SNES VI Function norm 2.508323951297e-02 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
>     0 KSP Residual norm 2.508323951297e-02
>     1 KSP Residual norm 1.744363683367e-05
>     2 KSP Residual norm 9.541678264164e-09
> Nonlinear solve did not converge due to DIVERGED_LOCAL_MIN iterations 1
> Solve Did NOT Converge!
> Solve failed... cutting timestep
> DT: 1.442732e+03
>  Solving time step 48, time=404699.00...
>   Initial |residual|_2 = 1.536061e-01
>   NL step  0, |residual|_2 = 1.536061e-01
>   0 SNES VI Function norm 1.536061428302e-01 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
>     0 KSP Residual norm 1.536061428302e-01
>     1 KSP Residual norm 7.144446293019e-05
>     2 KSP Residual norm 2.783195631751e-07
>   NL step  1, |residual|_2 = 2.513316e-02
>   1 SNES VI Function norm 2.513315575633e-02 Active lower constraints 0/2 upper constraints 0/0 Percent of total 0 Percent of bounded 0
>     0 KSP Residual norm 2.513315575633e-02
>     1 KSP Residual norm 2.187002762768e-05
>     2 KSP Residual norm 1.996619992363e-08
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 1

Can you upgrade to petsc-3.4?  (Or the 'maint' branch; 'master' is
reliable and newer, but has a couple interface changes that may or may
not affect your overall code.)  Use -snes_linesearch_monitor to see what
is happening in the line search.  This version of PETSc has new line
searches that may help with convergence.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/eb32e59b/attachment.pgp>

From einar.sorheim at gmail.com  Wed Oct 23 08:48:14 2013
From: einar.sorheim at gmail.com (=?ISO-8859-1?Q?Einar_S=F8rheim?=)
Date: Wed, 23 Oct 2013 15:48:14 +0200
Subject: [petsc-users] Problem with GAMG elasticity
 case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <AE80D149-ED23-4A27-835E-B14208A4F5FC@lbl.gov>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
	<6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
	<CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>
	<AE80D149-ED23-4A27-835E-B14208A4F5FC@lbl.gov>
Message-ID: <CAOZzF6oHxBVX2QJcDO4cV=Lf6_pa5VSU5_63b1GZn-pqyWCjZQ@mail.gmail.com>

I tried running with the following options :
-info
-malloc
-malloc_log
-malloc_dump
-malloc_debug
-memory_info
-log_summary log_summary.txt
-log
but didn't get a lot wiser however in my case would it be more efficient to
keep the ksp solver object, if so should I have one object eg. for the
thermal problem and another one for the mechanical?
The output from the end of the run:
 Total wall clock time actually solving:     856.231474
[0] PetscFinalize(): PetscFinalize() called
[0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784
max tags = 2147483647
Summary of Memory Usage in PETSc
[0]Current space PetscMalloc()ed 45936, max space PetscMalloced()
1.13728e+009
[0]OS cannot compute process memory
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
-2080374784
[0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
[0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
[0] PetscFOpen(): Opening file Log.0
[0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm
-2080374784
[0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator
embedded in a user MPI_Comm -2080374784
[0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after
removing reference from inner PETSc comm to this outer comm
[0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784
[0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm
-2080374784
[0] Petsc_DelThreadComm(): Deleting thread communicator data in an MPI_Comm
-2080374784
[0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374784
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: PetscMallocDumpLog() called without call to
PetscMallocSetDumpLog() this is often due to
                      setting the option -malloc_log AFTER
PetscInitialize() with PetscOptionsInsert() or PetscOptionsInsertFile()!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by
admeinar Wed Oct 23 11:54:03 2013
[0]PETSC ERROR: Libraries linked from
/cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
[0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
[0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes
--with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
--with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
(x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
--with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
(x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscMallocDumpLog() line 652 in
src/sys/memory/D:\PROGRA~1\PETSC-~1.3\src\sys\memory\mtr.c
[0]PETSC ERROR: PetscFinalize() line 1185 in
src/sys/objects/D:\PROGRA~1\PETSC-~1.3\src\sys\objects\pinit.c


2013/10/19 Mark F. Adams <mfadams at lbl.gov>

> You can try running a problem that finishes with
>
> -malloc
> -malloc_debug
> -malloc_dump
>
> to see if there are any memory issues.  It looks like you are completely
> destroying the solver object each time and starting over.  Valgrind would
> be very useful here because it looks like memory is getting corrupted if
> the same code runs many times.
>
>
> On Oct 19, 2013, at 3:39 PM, Einar S?rheim <einar.sorheim at gmail.com>
> wrote:
>
> yes, I call the given code many times, what i have seen so far is that
> changing e.g. the order of destruction statements changes the time of
> failure,I will try to debug to get more details
>
>
> 2013/10/18 Mark F. Adams <mfadams at lbl.gov>
>
>> I would guess there is some memory corruption or leak someplace.  Try
>> running with -on_error_attach_debugger, and try to get some more
>> information.  Do you just repeat a call to the code below many times?  Is
>> this deterministic, does it fail on the same solve each time?
>>
>>
>> On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com>
>> wrote:
>>
>> Uncommenting  KSPSetFromOptions solved my intial problem, however I got
>> a new problem that occurs after many timesteps, in our code we use petsc
>> gamg as a solver for several problems(e.g. thermal and mechanical) at each
>> time step, I have a suspicion it might have something to do with cleanup
>> after solving, the rror appears after many calls to the petsc gamg solver
>> interface, the error message is the following:
>>
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
>> corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580
>> src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules -
>> Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by
>> admeinar Fri Oct 18 12:26:04 2013
>> [0]PETSC ERROR: Libraries linked from
>> /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
>> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes
>> --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort"
>> --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files
>> (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu
>> --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files
>> (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>>
>> The source code for the petsc interface:
>>
>>       subroutine
>> petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond,
>> i_Solver
>>      $     ,coords_,i_nodedof)
>> #include <finclude/petscsys.h>
>> #include <finclude/petscvec.h>
>> #include <finclude/petscmat.h>
>> #include <finclude/petscpc.h>
>> #include <finclude/petscksp.h>
>> #include <finclude/petscviewer.h>
>>
>>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
>>       integer i_rowstart_(*),i_columnindex_(*)
>>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
>>       integer, allocatable  :: i_indices_(:)
>> !     Variables:
>> !
>> !  A       - matrix that defines linear system
>> !  ksp    - KSP context
>> !  ksp     - KSP context
>> !  x, b, u - approx solution, RHS, exact solution vectors
>> !
>> !      implicit none
>>       Vec     x,u,b,vec_coords
>>       PC      pc
>>       Mat     A,F,Matnull
>>       KSP    ksp
>>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
>>       PetscInt Istart,Iend
>>       PetscInt nsteps,one, neg_one
>>       PetscErrorCode ierr
>>       PetscBool  flg
>>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
>>       PetscOffset i_x
>>       PetscViewer viewer
>>
>>
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Enter petsc"
>>
>>       n      = 1
>>       nsteps = 1
>>       one    = 1
>>       neg_one = 1
>>       i1 = 1
>>       imaxiter = 1200
>>
>>
>>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
>>       call MatSetType(A, MATSEQAIJ,ierr)
>>       call MatSetUp(A,ierr)
>>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
>> !  The matrix is partitioned by contiguous chunks of rows across the
>> !  processors.  Determine which rows of the matrix are locally owned.
>>
>>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
>>
>>
>>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
>>       do i=1,i_nodes
>> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
>> !         i_indices_(i+1) = i
>> !         write(*,*) "Start of loop2"
>>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
>> !            write(*,*) "Start of loop 3"
>>             v = r_elements_(j)
>> !            write(*,*) i1,i,i1,j-1,v
>>             call
>> MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
>>             if (ierr.gt.0) stop
>>          end do
>>       end do
>> !      write(*,*) "End of matrix fill"
>>
>> !  Assemble matrix, using the 2-step process:
>> !       MatAssemblyBegin(), MatAssemblyEnd()
>> !  Computations can be done while messages are in transition
>> !  by placing code between these two statements.
>>
>>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>>
>> !  Create parallel vectors.
>> !   - When using VecCreate(), the parallel partitioning of the vector
>> !     is determined by PETSc at runtime.
>> !   - Note: We form 1 vector from scratch and then duplicate as needed.
>> !      write(*,*) "Start of vector fill"
>>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
>>       call VecSetType(u, VECSEQ,ierr)
>>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
>> !      call VecSetFromOptions(u,ierr)
>>       call VecDuplicate(u,b,ierr)
>>       call VecDuplicate(b,x,ierr)
>>       do i=1,i_nodes
>>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
>>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
>>       enddo
>> !  Assemble vector
>>
>>        call VecAssemblyBegin(x,ierr)
>>        call VecAssemblyEnd(x,ierr)
>>        call VecAssemblyBegin(b,ierr)
>>        call VecAssemblyEnd(b,ierr)
>>        call PetscTime(t1,ierr)
>> !       write(*,*) "End of vec fill time spent :", t1-t0
>> !  Create linear solver context
>>
>>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>>
>>
>> !  Set operators. Here the matrix that defines the linear system
>> !  also serves as the preconditioning matrix.
>>
>>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
>>
>>
>>       call KSPGetPC(ksp,pc,ierr)
>>       tol = 1.e-20
>>       call
>> KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>>       select case (i_Precond)
>>       case ( 1 )
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       case ( 2 )
>>          call PCSetType(pc,PCILU,ierr)
>>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
>>          call PCFactorSetUpMatSolverPackage(pc,ierr);
>>          call PCFactorGetMatrix(pc,F);
>>          call MatSuperluSetILUDropTol(F,1.e-4);
>>       case ( 3 )
>>          call PCSetType(pc,PCGAMG,ierr)
>> !         call PCGAMGInitializePackage()
>> !         call PCCreate_GAMG(pc,ierr)
>>       case DEFAULT
>>          call PCSetType(pc,PCJACOBI,ierr)
>>       end select
>>
>> !      call PCSetType(pc,PCJACOBI,ierr)
>>       select case (i_solver)
>>       case ( 0 )
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       case DEFAULT
>>          call KSPSetType(ksp,KSPCG,ierr)
>>       end select
>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>>       if (i_nodedof==3) then
>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>>          call
>> VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>>          call MatSetNearNullSpace(A,Matnull,ierr)
>>          call MatNullSpaceDestroy(Matnull,ierr)
>>
>>          write(*,*) "Finish setting null space ierr =", ierr
>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>>
>>       end if
>> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL,
>> PETSC_NULL,ierr)
>> !  Set runtime options, e.g.,
>> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
>> !  These options will override those specified above as long as
>> !  KSPSetFromOptions() is called _after_ any other customization
>> !  routines.
>>
>>       call KSPSetFromOptions(ksp,ierr)
>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>       call KSPSetUp(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "Time setup", t0-t1
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> !                      Solve the linear system
>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>
>>       call KSPSolve(ksp,b,x,ierr)
>>       call PetscTime(t1,ierr)
>>       write(*,*) "end solve, time used:",t1-t0
>>       call KSPGetSolution(ksp,x,ierr)
>>       call VecGetArray(x,x_array_,i_x,ierr)
>>       do i=1,i_nodes
>>          r_x_(i) = x_array_(i_x + i)
>>       enddo
>>       call VecRestoreArray(x,x_array_,i_x,ierr)
>>
>>
>> !  Free work space.  All PETSc objects should be destroyed when they
>> !  are no longer needed.
>> !      call PCDestroy(pc,ierr)
>>
>>       call VecDestroy(u,ierr)
>>       call VecDestroy(x,ierr)
>>       call VecDestroy(b,ierr)
>>       call MatDestroy(A,ierr)
>>       if (i_nodedof==3) then
>>          call VecDestroy(vec_coords,ierr)
>>          call MatDestroy(Matnull,ierr)
>>       endif
>>       call KSPDestroy(ksp,ierr)
>>       call PetscTime(t0,ierr)
>>       write(*,*) "time celan up :", t0-t1
>>
>>       end
>>
>> --
>> Einar S?rheim
>> <Mail Attachment>
>>
>>
>>
>
>
> --
> Einar S?rheim
>
>
>


-- 
Einar S?rheim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/d10bfa62/attachment-0001.html>

From christophe.ortiz at ciemat.es  Wed Oct 23 09:05:06 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 23 Oct 2013 16:05:06 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<162579E7-A6AA-467C-A5C1-BDF13E262305@mcs.anl.gov>
	<CANBrw+7bF5XsEA4J4yUCEfS+ByHf3Gxxv2x7Z8Ob6H2xtNwQOQ@mail.gmail.com>
	<E1B56AD9-7997-4639-AB7C-80E144FB76D0@mcs.anl.gov>
	<CANBrw+4TcdisdDOYaJH1XkUv+eQ18Sjt115x5SAfO_a7_4e51g@mail.gmail.com>
	<747FF53D-0700-401E-BE6D-F69BC9AAA38E@mcs.anl.gov>
	<CANBrw+4QyVFp0HggnpjNuGe7hpdgYXmFoWds2cM2pBDoMUabwg@mail.gmail.com>
	<CANBrw+70x5mv2VJ-5k7-HOK4iS+oP+zHGD+U7S5qHX+7=kavzA@mail.gmail.com>
	<87y562hab7.fsf@mcs.anl.gov>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<B3141C2C-B889-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
Message-ID: <CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>

On Tue, Oct 22, 2013 at 5:36 PM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
>
> On Tue, Oct 22, 2013 at 3:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Oct 22, 2013 at 8:29 AM, Christophe Ortiz <
>> christophe.ortiz at ciemat.es> wrote:
>>
>>> Hi Barry,
>>>
>>> Thanks for suggestions. Here is the output.
>>> Before using the options you suggest I get:
>>>
>>>  TSAdapt 'basic': step 182 accepted t=0.341561   + 3.416e-02
>>> wlte=1.1e-09 family='arkimex' scheme=0:'3' dt=3.757e-02
>>>       TSAdapt 'basic': step 183 accepted t=0.375717   + 3.757e-02
>>> wlte=9.05e-14 family='arkimex' scheme=0:'3' dt=4.133e-02
>>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 4.133e-02
>>> retrying with dt=1.033e-02
>>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   + 1.033e-02
>>> retrying with dt=2.583e-03
>>> ...
>>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +2.212e-198
>>> retrying with dt=5.529e-199
>>>       TSAdapt 'basic': step 184 stage rejected t=0.413289   +5.529e-199
>>> retrying with dt=1.382e-199
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Floating point exception!
>>> [0]PETSC ERROR: Vec entry at local location 12 is not-a-number or
>>> infinite at end of function: Parameter number 3!
>>>
>>>
>>>  I re-checked my Jacobian several times, and I found few errors. But now
>>> I think it is ok.
>>> Then, with the options you suggest, it stops immediatly. Not even one
>>> timestep:
>>>
>>
>> Something is very wrong here. The Jacobian should at least have something
>> on the diagonal from the time derivative,
>> but the Laplacian (u, v) and q (w) also have diagonals.
>>
>>    Matt
>>
>>
Hi guys,

I finally found what was wrong. I used -mat_view to check each element of
the Jacobian. The structure was ok but there was some unexpected values.
Then I checked my code and found a mistake while assigning values to some
array (the diagonal block). It was cumulating values during the loop for
(row i) {}. Now, at the beginning of the loop I use PetscMemzero(array) to
reset the array.

Now it works much better with 1bee and linesearch bt. It converges quickly
to large times in few timesteps.
Thanks all for your help and sorry to bother you so much :-).

Nevertheless...I still observe some problem, some oscillations in the
solution, but in some extreme cases. It occurs with the following system,
when q is very large:

u_t - alpha u_xx + (k.u.v - q.w) = 0
v_t - alpha v_xx + (k.u.v - q.w) = 0
w_t - (k.u.v - q.w) = 0

I guess the problem becomes stiff.
I tried assuming that the reaction is in steady state (w_t=0) and modifying
the IJacobian accordingly, but it did not work.

 Any suggestion to circumvent that ?
Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/cede88eb/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 23 09:20:20 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 09:20:20 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <1665283.VQd2nzgDvj@rigel>
References: <3178358.Nb05eOjZJN@dhcp071>
	<9A54FFFE-FC6A-48D2-AD99-C80237225DAD@mcs.anl.gov>
	<1677196.9ZmfKVC8up@dhcp071> <1665283.VQd2nzgDvj@rigel>
Message-ID: <87hac8jcgr.fsf@mcs.anl.gov>

Juha J?ykk? <juhaj at iki.fi> writes:

>> I will test with 3.3 later today if I have the time, but I'm pretty sure
>> things were "better" there.
>
> I did happen to have one system still having 3.2 on it, so I tested that 
> instead of 3.3 and at least 3.2 seems to be a bit more conservative in 
> numbers. For the same $CMD as before, I get
>
> for x in 1 2 4 ; do mpirun -np $x python -c "$CMD"; done
> 21 MiB / 21868 kB
> 21 MiB / 21888 kB
> 218 MiB / 224028 kB
> 22 MiB / 22852 kB
> 22 MiB / 22600 kB
> 22 MiB / 22620 kB
> 22 MiB / 22872 kB
> 79 MiB / 81640 kB
> 79 MiB / 81372 kB
> 22 MiB / 22628 kB
> 22 MiB / 22648 kB
> 22 MiB / 22632 kB
> 22 MiB / 22652 kB
> 22 MiB / 22880 kB
> 22 MiB / 22900 kB
> 22 MiB / 22908 kB
> 22 MiB / 22928 kB
> 51 MiB / 52944 kB
> 51 MiB / 52936 kB
> 51 MiB / 53188 kB
> 51 MiB / 53216 kB
>
> For comparison, 3.4.2 gives (from the previous email): 354 MiB for 1
> rank, 141 for 2 ranks and 81 for 4 ranks, which is a LOT less. I
> suspect this might have something to do with the DA -> DMDA change?

Hmm, I wonder where you're getting your numbers from.  

CMD='import psutil,os; rss = lambda: psutil.Process(os.getpid()).get_memory_info().rss; print rss()/2**20, "MiB initial"; from petsc4py import PETSc; print rss()/2**20, "MiB loaded petsc4py"; da = PETSc.DA().create(sizes=[100,100,100], proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE], boundary_type=[3,0,0], stencil_type=PETSc.DA.StencilType.BOX, dof=7, stencil_width=1, comm=PETSc.COMM_WORLD); print rss()/2**20, "MiB DMDA"'

Here is what I have with v3.4.2, which is almost exactly the same as you
show above for v3.2.  In particular, it is much better than your email
claimed for v3.4.2.

  $ mpirun.hydra -np 1 python2 -c "$CMD"
  8 MiB /
  29 MiB
  226 MiB
  $ mpirun.hydra -np 2 python2 -c "$CMD"
  8 MiB initial
  8 MiB initial
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  77 MiB DMDA
  77 MiB DMDA
  $ mpirun.hydra -np 4 python2 -c "$CMD"
  8 MiB initial
  8 MiB initial
  8 MiB initial
  8 MiB initial
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  53 MiB DMDA
  53 MiB DMDA
  53 MiB DMDA
  53 MiB DMDA

and with Barry's patch:

  $ mpirun.hydra -np 1 python2 -c "$CMD"
  8 MiB initial
  29 MiB loaded petsc4py
  172 MiB DMDA
  $ mpirun.hydra -np 2 python2 -c "$CMD"
  8 MiB initial
  8 MiB initial
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  51 MiB DMDA
  51 MiB DMDA
  $ mpirun.hydra -np 4 python2 -c "$CMD"
  8 MiB initial
  8 MiB initial
  8 MiB initial
  8 MiB initial
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  27 MiB loaded petsc4py
  39 MiB DMDA
  39 MiB DMDA
  38 MiB DMDA
  39 MiB DMDA


The psutil package is in PyPI and most distributions.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/b1390f8f/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 23 09:52:58 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 09:52:58 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<B3141C2C-B8 89-4BB9-844E-CC4085E3D9A2@mcs.anl.gov>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
Message-ID: <87sivshwdx.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> I finally found what was wrong. I used -mat_view to check each element of
> the Jacobian. The structure was ok but there was some unexpected values.
> Then I checked my code and found a mistake while assigning values to some
> array (the diagonal block). It was cumulating values during the loop for
> (row i) {}. Now, at the beginning of the loop I use PetscMemzero(array) to
> reset the array.
>
> Now it works much better with 1bee and linesearch bt. It converges quickly
> to large times in few timesteps.

That's reassuring to hear, thanks.

> Nevertheless...I still observe some problem, some oscillations in the
> solution, but in some extreme cases. It occurs with the following system,
> when q is very large:
>
> u_t - alpha u_xx + (k.u.v - q.w) = 0
> v_t - alpha v_xx + (k.u.v - q.w) = 0
> w_t - (k.u.v - q.w) = 0
>
> I guess the problem becomes stiff.

Does a shorter time step fix the oscillations?  Is this with
-ts_arkimex_type 1bee or something else?

Can you try -ts_arkimex_fully_implicit and add -snes_mf_operator if
necessary to get SNES to converge?  (I'm assuming you have used an IMEX
formulation here, but perhaps you already use fully implicit?)

> I tried assuming that the reaction is in steady state (w_t=0) and modifying
> the IJacobian accordingly, but it did not work.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/fea45aba/attachment.pgp>

From christophe.ortiz at ciemat.es  Wed Oct 23 10:23:00 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 23 Oct 2013 17:23:00 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87sivshwdx.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<CANBrw+7s_jcnCHhadw8t7zQu847YKKK5stH9_ez4jmE7f+rwjQ@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CAMYG4GkCaS-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
Message-ID: <CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>

On Wed, Oct 23, 2013 at 4:52 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > I finally found what was wrong. I used -mat_view to check each element of
> > the Jacobian. The structure was ok but there was some unexpected values.
> > Then I checked my code and found a mistake while assigning values to some
> > array (the diagonal block). It was cumulating values during the loop for
> > (row i) {}. Now, at the beginning of the loop I use PetscMemzero(array)
> to
> > reset the array.
> >
> > Now it works much better with 1bee and linesearch bt. It converges
> quickly
> > to large times in few timesteps.
>
> That's reassuring to hear, thanks.
>
> > Nevertheless...I still observe some problem, some oscillations in the
> > solution, but in some extreme cases. It occurs with the following system,
> > when q is very large:
> >
> > u_t - alpha u_xx + (k.u.v - q.w) = 0
> > v_t - alpha v_xx + (k.u.v - q.w) = 0
> > w_t - (k.u.v - q.w) = 0
> >
> > I guess the problem becomes stiff.
>
> Does a shorter time step fix the oscillations?


Hi Jed,

Thanks for the prompt reply.

No, it did not in this case. However, I noticed for "smoother" cases that
max timestep matters. I try to fix it to finaltime/100 to avoid too large
timesteps. I also use -ts_adapt_basic_clip 0.1,1.1 to avoid large timesteps.


>  Is this with
> -ts_arkimex_type 1bee or something else?
>

It occurs with 1bee, a2 or arkimex 3.


>
> Can you try -ts_arkimex_fully_implicit and add -snes_mf_operator if
> necessary to get SNES to converge?



I tried -ts_arkimex_fully_implicit and it gave a wrong result. As if there
was no diffusion. Seems there is an artefact with fullyimplicit option.

With -snes_mf_operator I got an error message:
[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Mat type mffd!



>  (I'm assuming you have used an IMEX
> formulation here, but perhaps you already use fully implicit?)
>

I put everything under IFunction and IJacobian. This part is not clear to
me actually. I understand that in IMEX methods, the LHS is solved
implicitely and the RHS explicitely. What happens when I use an IMEX method
with no RHS, everything in the LHS ? Is there any explicit stage ?

Actually, I could solve the problem by adding mesh points. Since I start
from steep gaussian distributions with large peak values, maybe there was a
problem with large gradients.

BTW, is it possible to have adaptive mesh in 1D with PETSc ? I am thinking
of steep profiles that evolve and that require a fine mesh at the beginning.

Christophe


>
> > I tried assuming that the reaction is in steady state (w_t=0) and
> modifying
> > the IJacobian accordingly, but it did not work.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/5d4b86e0/attachment-0001.html>

From cedric.doucet at inria.fr  Wed Oct 23 10:23:19 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Wed, 23 Oct 2013 17:23:19 +0200 (CEST)
Subject: [petsc-users] Partitioning meshes with DMPlex
In-Reply-To: <529424868.50817.1382541411979.JavaMail.root@inria.fr>
Message-ID: <1245797056.50910.1382541799921.JavaMail.root@inria.fr>


Hello, 

what is the best way to partition unstructured meshes stored with DMPlex? 
Does one have to create an adjacency matrix and use MatPartitioning routines? 
I do not want to partition the mesh graph (by cutting edges) but having a true domain decomposition (sets of cells per processor). 

Thank you very much for your help! 

Best regards, 

C?dric Doucet 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/133273e5/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 23 10:39:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 10:39:09 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CAMYG4GkCaS
	-m2tu+bThzVVS4URHdZ9GV=u2pX5Nw26KHGuT0Ug@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
Message-ID: <87hac8hu8y.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

>> Does a shorter time step fix the oscillations?
>
>
> Hi Jed,
>
> Thanks for the prompt reply.
>
> No, it did not in this case. However, I noticed for "smoother" cases that
> max timestep matters. I try to fix it to finaltime/100 to avoid too large
> timesteps. I also use -ts_adapt_basic_clip 0.1,1.1 to avoid large timesteps.

This just slows the rate of increasing time step.

>>  Is this with
>> -ts_arkimex_type 1bee or something else?
>>
>
> It occurs with 1bee, a2 or arkimex 3.

Stick with 1bee until we understand better.

>> Can you try -ts_arkimex_fully_implicit and add -snes_mf_operator if
>> necessary to get SNES to converge?
>
>
>
> I tried -ts_arkimex_fully_implicit and it gave a wrong result. As if there
> was no diffusion. Seems there is an artefact with fullyimplicit option.

That does not sound right.  Can you send a reproducible test case (best
as a patch to PETSc)?

> With -snes_mf_operator I got an error message:
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Mat type mffd!

ALWAYS, ALWAYS, ALWAYS send the ENTIRE error message!

(Sorry about shouting, but I have to type this many times per day.
Truncating error messages kills kittens!)

>>  (I'm assuming you have used an IMEX
>> formulation here, but perhaps you already use fully implicit?)
>>
>
> I put everything under IFunction and IJacobian. This part is not clear to
> me actually. I understand that in IMEX methods, the LHS is solved
> implicitely and the RHS explicitely. What happens when I use an IMEX method
> with no RHS, everything in the LHS ? Is there any explicit stage ?

No, if the RHS is empty, it should reduce to an implicit method.

> Actually, I could solve the problem by adding mesh points. Since I start
> from steep gaussian distributions with large peak values, maybe there was a
> problem with large gradients.
>
> BTW, is it possible to have adaptive mesh in 1D with PETSc ? I am thinking
> of steep profiles that evolve and that require a fine mesh at the beginning.

You can do moving meshes (r-adaptivity) by writing coordinates as a
coupled system.  Or you can create a new grid and interpolate.  You can
use DMRefine and DMCoarsen.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/c646a1e6/attachment.pgp>

From jedbrown at mcs.anl.gov  Wed Oct 23 10:43:55 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 10:43:55 -0500
Subject: [petsc-users] Partitioning meshes with DMPlex
In-Reply-To: <1245797056.50910.1382541799921.JavaMail.root@inria.fr>
References: <1245797056.50910.1382541799921.JavaMail.root@inria.fr>
Message-ID: <878uxkhu10.fsf@mcs.anl.gov>

Cedric Doucet <cedric.doucet at inria.fr> writes:

> Hello, 
>
> what is the best way to partition unstructured meshes stored with DMPlex? 

You can use DMPlexCreatePartition, which supports metis and chaco.

Matt, DMPlexCreatePartition documentation has left out the partitioner "name".

> Does one have to create an adjacency matrix and use MatPartitioning routines? 
> I do not want to partition the mesh graph (by cutting edges) but having a true domain decomposition (sets of cells per processor). 
>
> Thank you very much for your help! 
>
> Best regards, 
>
> C?dric Doucet 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/96ba2b48/attachment.pgp>

From christophe.ortiz at ciemat.es  Wed Oct 23 11:55:00 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Wed, 23 Oct 2013 18:55:00 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87hac8hu8y.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<CANBrw+4nDieNxbPd+MW0a5iqUhacW3B-ufvkjp2eMROXn027Uw@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<CANBrw+67guKEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
	<87hac8hu8y.fsf@mcs.anl.gov>
Message-ID: <CANBrw+4UTHi-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>

On Wed, Oct 23, 2013 at 5:39 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> >> Does a shorter time step fix the oscillations?
> >
> >
> > Hi Jed,
> >
> > Thanks for the prompt reply.
> >
> > No, it did not in this case. However, I noticed for "smoother" cases that
> > max timestep matters. I try to fix it to finaltime/100 to avoid too large
> > timesteps. I also use -ts_adapt_basic_clip 0.1,1.1 to avoid large
> timesteps.
>
> This just slows the rate of increasing time step.
>
> >>  Is this with
> >> -ts_arkimex_type 1bee or something else?
> >>
> >
> > It occurs with 1bee, a2 or arkimex 3.
>
> Stick with 1bee until we understand better.
>

Ok.


>
> >> Can you try -ts_arkimex_fully_implicit and add -snes_mf_operator if
> >> necessary to get SNES to converge?
> >
> >
> >
> > I tried -ts_arkimex_fully_implicit and it gave a wrong result. As if
> there
> > was no diffusion. Seems there is an artefact with fullyimplicit option.
>
> That does not sound right.  Can you send a reproducible test case (best
> as a patch to PETSc)?
>

What do you mean by patch to PETSc ? How do I do that ?


>
> > With -snes_mf_operator I got an error message:
> > [0]PETSC ERROR: No support for this operation for this object type!
> > [0]PETSC ERROR: Mat type mffd!
>
> ALWAYS, ALWAYS, ALWAYS send the ENTIRE error message!
>
> (Sorry about shouting, but I have to type this many times per day.
> Truncating error messages kills kittens!)
>


Sorry about that. Here is the entire error message:

[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Mat type mffd!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
mazinger.ciemat.es by u5751 Wed Oct 23 12:46:04 2013
[0]PETSC ERROR: Libraries linked from
/home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
[0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
[0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
--with-cc=icc --with-debugging=1 --with-scalar-type=real
--with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatZeroEntries() line 5189 in src/mat/interface/matrix.c
[0]PETSC ERROR: FormIJacobian() line 689 in
src/ts/examples/tutorials/diffusion.c
[0]PETSC ERROR: TSComputeIJacobian() line 718 in src/ts/interface/ts.c
[0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 995 in
src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: SNESTSFormJacobian() line 3397 in src/ts/interface/ts.c
[0]PETSC ERROR: SNESComputeJacobian() line 2152 in src/snes/interface/snes.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 218 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
[0]PETSC ERROR: main() line 457 in src/ts/examples/tutorials/diffusion.c
./compile_diffusion: line 17: 20506 Aborted                 ./diffusion_c
-ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
-ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
-snes_linesearch_type bt -snes_mf_operator



>
> >>  (I'm assuming you have used an IMEX
> >> formulation here, but perhaps you already use fully implicit?)
> >>
> >
> > I put everything under IFunction and IJacobian. This part is not clear to
> > me actually. I understand that in IMEX methods, the LHS is solved
> > implicitely and the RHS explicitely. What happens when I use an IMEX
> method
> > with no RHS, everything in the LHS ? Is there any explicit stage ?
>
> No, if the RHS is empty, it should reduce to an implicit method.
>
> > Actually, I could solve the problem by adding mesh points. Since I start
> > from steep gaussian distributions with large peak values, maybe there
> was a
> > problem with large gradients.
> >
> > BTW, is it possible to have adaptive mesh in 1D with PETSc ? I am
> thinking
> > of steep profiles that evolve and that require a fine mesh at the
> beginning.
>
> You can do moving meshes (r-adaptivity) by writing coordinates as a
> coupled system.  Or you can create a new grid and interpolate.  You can
> use DMRefine and DMCoarsen.
>

Great. I will check this out.

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/f7d8860d/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 23 12:14:27 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 12:14:27 -0500
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4UTHi-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<CANBrw+67gu
	KEyXwE390YboYFTL7VHEs6koCkOhv8BvYdStOXqw@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
	<87hac8hu8y.fsf@mcs.anl.gov>
	<CANBrw+4UTHi-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>
Message-ID: <87mwlzhpu4.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

>> That does not sound right.  Can you send a reproducible test case (best
>> as a patch to PETSc)?
>>
>
> What do you mean by patch to PETSc ? How do I do that ?

See "contributing a small patch" (or any of the other workflows).

https://bitbucket.org/petsc/petsc/wiki/Home

> Sorry about that. Here is the entire error message:
>
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Mat type mffd!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
> mazinger.ciemat.es by u5751 Wed Oct 23 12:46:04 2013
> [0]PETSC ERROR: Libraries linked from
> /home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
> [0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
> [0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
> --with-cc=icc --with-debugging=1 --with-scalar-type=real
> --with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatZeroEntries() line 5189 in src/mat/interface/matrix.c
> [0]PETSC ERROR: FormIJacobian() line 689 in
> src/ts/examples/tutorials/diffusion.c

This is your code calling MatZeroEntries on the Amat (first Mat
argument).  The convention is to assemble into Pmat (the preconditioning
matrix; second Mat argument).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/66a0ca3b/attachment.pgp>

From bsmith at mcs.anl.gov  Wed Oct 23 14:23:19 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 23 Oct 2013 14:23:19 -0500
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAOZzF6oHxBVX2QJcDO4cV=Lf6_pa5VSU5_63b1GZn-pqyWCjZQ@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
	<6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
	<CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>
	<AE80D149-ED23-4A27-835E-B14208A4F5FC@lbl.gov>
	<CAOZzF6oHxBVX2QJcDO4cV=Lf6_pa5VSU5_63b1GZn-pqyWCjZQ@mail.gmail.com>
Message-ID: <83ED04D2-DE73-4A70-BDEE-0994317E0051@mcs.anl.gov>


   Don?t run with the -malloc_log Mark did not suggest using that option.


On Oct 23, 2013, at 8:48 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:

> I tried running with the following options :
> -info 
> -malloc 
> -malloc_log
> -malloc_dump 
> -malloc_debug 
> -memory_info 
> -log_summary log_summary.txt 
> -log
> but didn't get a lot wiser however in my case would it be more efficient to keep the ksp solver object, if so should I have one object eg. for the thermal problem and another one for the mechanical? 
> The output from the end of the run:
>  Total wall clock time actually solving:     856.231474
> [0] PetscFinalize(): PetscFinalize() called
> [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647
> Summary of Memory Usage in PETSc
> [0]Current space PetscMalloc()ed 45936, max space PetscMalloced() 1.13728e+009
> [0]OS cannot compute process memory
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> [0] PetscFOpen(): Opening file Log.0
> [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374784
> [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm
> [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784
> [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> [0] Petsc_DelThreadComm(): Deleting thread communicator data in an MPI_Comm -2080374784
> [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374784
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: PetscMallocDumpLog() called without call to PetscMallocSetDumpLog() this is often due to
>                       setting the option -malloc_log AFTER PetscInitialize() with PetscOptionsInsert() or PetscOptionsInsertFile()!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Wed Oct 23 11:54:03 2013
> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocDumpLog() line 652 in src/sys/memory/D:\PROGRA~1\PETSC-~1.3\src\sys\memory\mtr.c
> [0]PETSC ERROR: PetscFinalize() line 1185 in src/sys/objects/D:\PROGRA~1\PETSC-~1.3\src\sys\objects\pinit.c
> 
> 
> 2013/10/19 Mark F. Adams <mfadams at lbl.gov>
> You can try running a problem that finishes with
> 
> -malloc
> -malloc_debug
> -malloc_dump
> 
> to see if there are any memory issues.  It looks like you are completely destroying the solver object each time and starting over.  Valgrind would be very useful here because it looks like memory is getting corrupted if the same code runs many times.
> 
> 
> On Oct 19, 2013, at 3:39 PM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> 
>> yes, I call the given code many times, what i have seen so far is that changing e.g. the order of destruction statements changes the time of failure,I will try to debug to get more details
>> 
>> 
>> 2013/10/18 Mark F. Adams <mfadams at lbl.gov>
>> I would guess there is some memory corruption or leak someplace.  Try running with -on_error_attach_debugger, and try to get some more information.  Do you just repeat a call to the code below many times?  Is this deterministic, does it fail on the same solve each time? 
>> 
>> 
>> On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
>> 
>>> Uncommenting  KSPSetFromOptions solved my intial problem, however I got a new problem that occurs after many timesteps, in our code we use petsc gamg as a solver for several problems(e.g. thermal and mechanical) at each time step, I have a suspicion it might have something to do with cleanup after solving, the rror appears after many calls to the petsc gamg solver interface, the error message is the following:
>>> 
>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580 src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Signal received!
>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Fri Oct 18 12:26:04 2013
>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
>>> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
>>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
>>> 
>>> The source code for the petsc interface: 
>>> 
>>>       subroutine petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond, i_Solver
>>>      $     ,coords_,i_nodedof)
>>> #include <finclude/petscsys.h>
>>> #include <finclude/petscvec.h>
>>> #include <finclude/petscmat.h>
>>> #include <finclude/petscpc.h>
>>> #include <finclude/petscksp.h>
>>> #include <finclude/petscviewer.h>
>>> 
>>>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
>>>       integer i_rowstart_(*),i_columnindex_(*)
>>>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
>>>       integer, allocatable  :: i_indices_(:)
>>> !     Variables:
>>> !
>>> !  A       - matrix that defines linear system
>>> !  ksp    - KSP context
>>> !  ksp     - KSP context
>>> !  x, b, u - approx solution, RHS, exact solution vectors
>>> !
>>> !      implicit none
>>>       Vec     x,u,b,vec_coords
>>>       PC      pc
>>>       Mat     A,F,Matnull
>>>       KSP    ksp
>>>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
>>>       PetscInt Istart,Iend
>>>       PetscInt nsteps,one, neg_one
>>>       PetscErrorCode ierr
>>>       PetscBool  flg
>>>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
>>>       PetscOffset i_x
>>>       PetscViewer viewer
>>> 
>>> 
>>>       call PetscTime(t0,ierr)
>>>       write(*,*) "Enter petsc"
>>> 
>>>       n      = 1
>>>       nsteps = 1
>>>       one    = 1
>>>       neg_one = 1
>>>       i1 = 1
>>>       imaxiter = 1200
>>> 
>>> 
>>>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
>>>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
>>>       call MatSetType(A, MATSEQAIJ,ierr)
>>>       call MatSetUp(A,ierr)
>>>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
>>> !  The matrix is partitioned by contiguous chunks of rows across the
>>> !  processors.  Determine which rows of the matrix are locally owned. 
>>> 
>>>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
>>> 
>>> 
>>>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
>>>       do i=1,i_nodes
>>> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
>>> !         i_indices_(i+1) = i
>>> !         write(*,*) "Start of loop2"
>>>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
>>> !            write(*,*) "Start of loop 3"
>>>             v = r_elements_(j)
>>> !            write(*,*) i1,i,i1,j-1,v
>>>             call MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
>>>             if (ierr.gt.0) stop
>>>          end do
>>>       end do
>>> !      write(*,*) "End of matrix fill"
>>> 
>>> !  Assemble matrix, using the 2-step process:
>>> !       MatAssemblyBegin(), MatAssemblyEnd()
>>> !  Computations can be done while messages are in transition
>>> !  by placing code between these two statements.
>>> 
>>>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
>>>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
>>> 
>>> !  Create parallel vectors.
>>> !   - When using VecCreate(), the parallel partitioning of the vector
>>> !     is determined by PETSc at runtime.
>>> !   - Note: We form 1 vector from scratch and then duplicate as needed.
>>> !      write(*,*) "Start of vector fill"
>>>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
>>>       call VecSetType(u, VECSEQ,ierr)
>>>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
>>> !      call VecSetFromOptions(u,ierr)      
>>>       call VecDuplicate(u,b,ierr)
>>>       call VecDuplicate(b,x,ierr)
>>>       do i=1,i_nodes
>>>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
>>>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
>>>       enddo
>>> !  Assemble vector
>>> 
>>>        call VecAssemblyBegin(x,ierr)
>>>        call VecAssemblyEnd(x,ierr)
>>>        call VecAssemblyBegin(b,ierr)
>>>        call VecAssemblyEnd(b,ierr)
>>>        call PetscTime(t1,ierr)
>>> !       write(*,*) "End of vec fill time spent :", t1-t0
>>> !  Create linear solver context
>>> 
>>>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
>>> 
>>> 
>>> !  Set operators. Here the matrix that defines the linear system
>>> !  also serves as the preconditioning matrix.
>>> 
>>>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
>>> 
>>> 
>>>       call KSPGetPC(ksp,pc,ierr)
>>>       tol = 1.e-20
>>>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
>>>       select case (i_Precond)
>>>       case ( 1 )
>>>          call PCSetType(pc,PCJACOBI,ierr)
>>>       case ( 2 )
>>>          call PCSetType(pc,PCILU,ierr)
>>>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
>>>          call PCFactorSetUpMatSolverPackage(pc,ierr); 
>>>          call PCFactorGetMatrix(pc,F);
>>>          call MatSuperluSetILUDropTol(F,1.e-4);
>>>       case ( 3 )
>>>          call PCSetType(pc,PCGAMG,ierr)
>>> !         call PCGAMGInitializePackage()
>>> !         call PCCreate_GAMG(pc,ierr)
>>>       case DEFAULT
>>>          call PCSetType(pc,PCJACOBI,ierr)
>>>       end select
>>> 
>>> !      call PCSetType(pc,PCJACOBI,ierr)
>>>       select case (i_solver)
>>>       case ( 0 )
>>>          call KSPSetType(ksp,KSPCG,ierr)
>>>       case DEFAULT
>>>          call KSPSetType(ksp,KSPCG,ierr)
>>>       end select
>>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
>>>       if (i_nodedof==3) then
>>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
>>>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
>>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
>>>          call MatSetNearNullSpace(A,Matnull,ierr)
>>>          call MatNullSpaceDestroy(Matnull,ierr)
>>> 
>>>          write(*,*) "Finish setting null space ierr =", ierr
>>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
>>> 
>>>       end if
>>> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL, PETSC_NULL,ierr)
>>> !  Set runtime options, e.g.,
>>> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
>>> !  These options will override those specified above as long as
>>> !  KSPSetFromOptions() is called _after_ any other customization
>>> !  routines.
>>> 
>>>       call KSPSetFromOptions(ksp,ierr)
>>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
>>>       call KSPSetUp(ksp,ierr)
>>>       call PetscTime(t0,ierr)
>>>       write(*,*) "Time setup", t0-t1
>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>>> !                      Solve the linear system
>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
>>> 
>>>       call KSPSolve(ksp,b,x,ierr)
>>>       call PetscTime(t1,ierr)
>>>       write(*,*) "end solve, time used:",t1-t0
>>>       call KSPGetSolution(ksp,x,ierr)
>>>       call VecGetArray(x,x_array_,i_x,ierr)
>>>       do i=1,i_nodes
>>>          r_x_(i) = x_array_(i_x + i)
>>>       enddo
>>>       call VecRestoreArray(x,x_array_,i_x,ierr)
>>> 
>>> 
>>> !  Free work space.  All PETSc objects should be destroyed when they
>>> !  are no longer needed.
>>> !      call PCDestroy(pc,ierr)
>>> 
>>>       call VecDestroy(u,ierr)
>>>       call VecDestroy(x,ierr)
>>>       call VecDestroy(b,ierr)
>>>       call MatDestroy(A,ierr)
>>>       if (i_nodedof==3) then
>>>          call VecDestroy(vec_coords,ierr)
>>>          call MatDestroy(Matnull,ierr)
>>>       endif
>>>       call KSPDestroy(ksp,ierr)
>>>       call PetscTime(t0,ierr)
>>>       write(*,*) "time celan up :", t0-t1
>>> 
>>>       end
>>> 
>>> -- 
>>> Einar S?rheim
>>> <Mail Attachment>
>> 
>> 
>> 
>> 
>> -- 
>> Einar S?rheim
> 
> 
> 
> 
> -- 
> Einar S?rheim


From jedbrown at mcs.anl.gov  Wed Oct 23 17:22:29 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 17:22:29 -0500
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <CAN5Wd-L7dK4jPj_Tj40=pLwwx-EwcztWxBWC5_vJS3YEtKPCrA@mail.gmail.com>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
	<DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
	<CAN5Wd-L7dK4jPj_Tj40=pLwwx-EwcztWxBWC5_vJS3YEtKPCrA@mail.gmail.com>
Message-ID: <871u3bfx0a.fsf@mcs.anl.gov>

Fande Kong <fd.kong at siat.ac.cn> writes:

> Hi Barry,
>
> I contacted the supercomputer center, and they asked me for a test case so
> that they can forward it to IBM. Is it possible that we write a test case
> only using MPI?  It is not a good idea that we send them the whole petsc
> code and my code.

This may be possible, but this smells of a message ordering/rendezvous
problem, in which case you'll have to reproduce essentially the same
communication pattern.  The fact that you don't see the error sooner in
your program execution (and that it doesn't affect lots of other people)
indicates that the bug may be very specific to your communication
pattern.  In fact, it is likely that changing your element distribution
algorithm, or some similar changes, may make the bug vanish.
Reproducing all this matching context in a stand-alone code is likely to
be quite a lot of effort.

I would configure the system to dump core on errors and package up the
test case.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/7686d8d8/attachment.pgp>

From knepley at gmail.com  Wed Oct 23 17:25:40 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 23 Oct 2013 17:25:40 -0500
Subject: [petsc-users] Partitioning meshes with DMPlex
In-Reply-To: <878uxkhu10.fsf@mcs.anl.gov>
References: <1245797056.50910.1382541799921.JavaMail.root@inria.fr>
	<878uxkhu10.fsf@mcs.anl.gov>
Message-ID: <CAMYG4GmVtj1EXn_cAdd43wwB0v24p1_qp3zQV5v3iOYrecu0yQ@mail.gmail.com>

On Wed, Oct 23, 2013 at 10:43 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Cedric Doucet <cedric.doucet at inria.fr> writes:
>
> > Hello,
> >
> > what is the best way to partition unstructured meshes stored with DMPlex?
>
> You can use DMPlexCreatePartition, which supports metis and chaco.
>
> Matt, DMPlexCreatePartition documentation has left out the partitioner
> "name".


I assume what you really want is DMPlexDistribute().

   Matt


> > Does one have to create an adjacency matrix and use MatPartitioning
> routines?
> > I do not want to partition the mesh graph (by cutting edges) but having
> a true domain decomposition (sets of cells per processor).
> >
> > Thank you very much for your help!
> >
> > Best regards,
> >
> > C?dric Doucet
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/d37c3b5f/attachment-0001.html>

From fd.kong at siat.ac.cn  Wed Oct 23 18:22:20 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Wed, 23 Oct 2013 17:22:20 -0600
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <871u3bfx0a.fsf@mcs.anl.gov>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
	<DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
	<CAN5Wd-L7dK4jPj_Tj40=pLwwx-EwcztWxBWC5_vJS3YEtKPCrA@mail.gmail.com>
	<871u3bfx0a.fsf@mcs.anl.gov>
Message-ID: <CAN5Wd-+h5QpbEKiEvyx0K_wbEWrmzruTG3-dgXmYYeT64G9Wow@mail.gmail.com>

Jed,

Thank you very much.

They made some observations, and they might make some progresses.  I at
least can make some runs now. They also say that it is something about
ordering/rendezvous. They said that there may be too many messages or too
long messages or both.




On Wed, Oct 23, 2013 at 4:22 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Fande Kong <fd.kong at siat.ac.cn> writes:
>
> > Hi Barry,
> >
> > I contacted the supercomputer center, and they asked me for a test case
> so
> > that they can forward it to IBM. Is it possible that we write a test case
> > only using MPI?  It is not a good idea that we send them the whole petsc
> > code and my code.
>
> This may be possible, but this smells of a message ordering/rendezvous
> problem, in which case you'll have to reproduce essentially the same
> communication pattern.  The fact that you don't see the error sooner in
> your program execution (and that it doesn't affect lots of other people)
> indicates that the bug may be very specific to your communication
> pattern.  In fact, it is likely that changing your element distribution
> algorithm, or some similar changes, may make the bug vanish.
> Reproducing all this matching context in a stand-alone code is likely to
> be quite a lot of effort.
>
> I would configure the system to dump core on errors and package up the
> test case.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/4065764f/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 23 18:33:21 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 Oct 2013 18:33:21 -0500
Subject: [petsc-users] How to understand these error messages
In-Reply-To: <CAN5Wd-+h5QpbEKiEvyx0K_wbEWrmzruTG3-dgXmYYeT64G9Wow@mail.gmail.com>
References: <87ip11lzpf.fsf@mcs.anl.gov>
	<1488219718.8188107.1372258561902.JavaMail.root@alcf.anl.gov>
	<CAN5Wd-J67pQksXEaOPy7HumtLWK0_nV__mkfHOw1zSh3rAW03A@mail.gmail.com>
	<DC2E5B82-7799-43C0-A6C7-5314CD46F484@mcs.anl.gov>
	<CAN5Wd-L7dK4jPj_Tj40=pLwwx-EwcztWxBWC5_vJS3YEtKPCrA@mail.gmail.com>
	<871u3bfx0a.fsf@mcs.anl.gov>
	<CAN5Wd-+h5QpbEKiEvyx0K_wbEWrmzruTG3-dgXmYYeT64G9Wow@mail.gmail.com>
Message-ID: <87fvrref5q.fsf@mcs.anl.gov>

Fande Kong <fd.kong at siat.ac.cn> writes:
> They made some observations, and they might make some progresses.  I at
> least can make some runs now. They also say that it is something about
> ordering/rendezvous. They said that there may be too many messages or too
> long messages or both.

-log_summary for a slightly smaller problem will give a good idea of
 message size and count.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131023/7ef00eef/attachment.pgp>

From christophe.ortiz at ciemat.es  Thu Oct 24 05:09:37 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 24 Oct 2013 12:09:37 +0200
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <87mwlzhpu4.fsf@mcs.anl.gov>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
	<87hac8hu8y.fsf@mcs.anl.gov>
	<CANBrw+4UTHi-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>
	<87mwlzhpu4.fsf@mcs.anl.gov>
Message-ID: <CANBrw+4Gxp6Njv5+m2BzeYEcnL+1FbUgwWh3EV9+ZfFXcgTagg@mail.gmail.com>

On Wed, Oct 23, 2013 at 7:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> >> That does not sound right.  Can you send a reproducible test case (best
> >> as a patch to PETSc)?
> >>
> >
> > What do you mean by patch to PETSc ? How do I do that ?
>
> See "contributing a small patch" (or any of the other workflows).
>
> https://bitbucket.org/petsc/petsc/wiki/Home


Ok, I will read the instructions and try.


>
>
> > Sorry about that. Here is the entire error message:
> >
> > [0]PETSC ERROR: No support for this operation for this object type!
> > [0]PETSC ERROR: Mat type mffd!
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
> > mazinger.ciemat.es by u5751 Wed Oct 23 12:46:04 2013
> > [0]PETSC ERROR: Libraries linked from
> > /home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
> > [0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
> > [0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
> > --with-cc=icc --with-debugging=1 --with-scalar-type=real
> > --with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: MatZeroEntries() line 5189 in src/mat/interface/matrix.c
> > [0]PETSC ERROR: FormIJacobian() line 689 in
> > src/ts/examples/tutorials/diffusion.c
>
> This is your code calling MatZeroEntries on the Amat (first Mat
> argument).  The convention is to assemble into Pmat (the preconditioning
> matrix; second Mat argument).
>

I assemble first Pmat but in my code MatZeroEntries was used with Amat,
which PETSc did not appreciate...
I changed that and here is the output. While without these options it
converges--though giving solution with oscillations--now with
-ts_arkimex_fully_implicit -snes_mf_operator it does not converge at all.
No step is accepted, not even the first one:

      TSAdapt 'basic': step   0 stage rejected t=0          + 1.000e-12
retrying with dt=2.500e-13
      TSAdapt 'basic': step   0 stage rejected t=0          + 2.500e-13
retrying with dt=6.250e-14
      TSAdapt 'basic': step   0 stage rejected t=0          + 6.250e-14
retrying with dt=1.562e-14
      TSAdapt 'basic': step   0 stage rejected t=0          + 1.562e-14
retrying with dt=3.906e-15

...  ....
      TSAdapt 'basic': step   0 stage rejected t=0          +2.888e-287
retrying with dt=7.221e-288
      TSAdapt 'basic': step   0 stage rejected t=0          +7.221e-288
retrying with dt=1.805e-288
      TSAdapt 'basic': step   0 stage rejected t=0          +1.805e-288
retrying with dt=4.513e-289
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Vec entry at local location 1103 is not-a-number or
infinite at end of function: Parameter number 3!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
mazinger.ciemat.es by u5751 Thu Oct 24 03:37:56 2013
[0]PETSC ERROR: Libraries linked from
/home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
[0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
[0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
--with-cc=icc --with-debugging=1 --with-scalar-type=real
--with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecValidValues() line 30 in src/vec/vec/interface/rvector.c
[0]PETSC ERROR: SNESComputeFunction() line 1998 in src/snes/interface/snes.c
[0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
[0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
[0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
[0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
[0]PETSC ERROR: main() line 457 in src/ts/examples/tutorials/diffusion.c
./compile_diffusion: line 17: 23584 Aborted                 ./diffusion_c
-ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
-ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
-snes_linesearch_type bt -ts_arkimex_fully_implicit -snes_mf_operator

Using -snes_converged_reason and -ksp_converged_reason gives:

    Linear solve did not converge due to DIVERGED_DTOL iterations 30
  Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 0
      TSAdapt 'basic': step   0 stage rejected t=0          + 1.000e-12
retrying with dt=2.500e-13

Thus, I increased the maximum number of iterations of the KSP with
  ierr = SNESSetMaxLinearSolveFailures(snes,100);
and
  ierr =
KSPSetTolerances(ksp,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT,1000);

Then it converges, steps are accepted. However, the solution obtained is
weird, as if there was no diffusion. Moreover, values are incorrect. I
start with a gaussian peak at 1e19 and end up with a gaussian with a peak
at 1e25.

BTW, why using -ts_arkimex_fully_implicit if all the terms are in the F
(LHS) ? You said that by definition there is no explicit stage in that case.

However, I could obtain a good solution by adding mesh points (without
fully implicit and mf_operator). Maybe it was due to large gradients and
not enough points.

Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131024/bacaf5ad/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 24 06:42:07 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 24 Oct 2013 06:42:07 -0500
Subject: [petsc-users] Problem with GAMG elasticity
	case:'createdefaultdata' not set(?) need to support NULL data!
In-Reply-To: <CAOZzF6rDRFFbR97bpWqhkCsz5EUFOqoDMtCHZBYGOMQ1ngay9Q@mail.gmail.com>
References: <CAOZzF6p-Aq9d10d-xJnkT3cwyPYn53aQQojYqppKAha1rPJj0g@mail.gmail.com>
	<CAMYG4GkJ8hv5meiCJa05AK6+C9bZvxG4HjYRh8sOe_Wpcy3aJw@mail.gmail.com>
	<8EE4826E-4436-4FE4-84D9-2EBF4C877340@lbl.gov>
	<CAMYG4GkAqHNXeF2-cP50nBrr6HvpdRUg3d7pciJdY=DmF7PzRQ@mail.gmail.com>
	<1DD67C25-D646-4BD7-B44D-709D505CF267@lbl.gov>
	<87txggipud.fsf@mcs.anl.gov>
	<CAOZzF6qyW5_KqL8gWbSc43OYwrEN8YWBeL+jARMpE1JALvLw7g@mail.gmail.com>
	<6958C2D4-F7CA-43F7-8E56-A3933E259FA5@lbl.gov>
	<CAOZzF6qR0-GHtfhEPD-2i3OB09=ET+QfKFDfcLZ2Hg=OO8paCw@mail.gmail.com>
	<AE80D149-ED23-4A27-835E-B14208A4F5FC@lbl.gov>
	<CAOZzF6oHxBVX2QJcDO4cV=Lf6_pa5VSU5_63b1GZn-pqyWCjZQ@mail.gmail.com>
	<83ED04D2-DE73-4A70-BDEE-0994317E0051@mcs.anl.gov>
	<CAOZzF6rDRFFbR97bpWqhkCsz5EUFOqoDMtCHZBYGOMQ1ngay9Q@mail.gmail.com>
Message-ID: <71DB6D5F-F64E-4351-8B91-43D660E2E271@mcs.anl.gov>


  So apparently no memory leak.

   If you are still getting crashes you will need to run the case that crashes 1) with valgrind if you haven?t or 2) using -on_error_attach_debugger or -start_in_debugger to track down the cause of the crash.

   Barry



On Oct 24, 2013, at 2:09 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:

> Well i I did a rerun without -malloc_log, this is what I got (I have also attached the log-sumary file):
> 
> [0] PetscFinalize(): PetscFinalize() called
> [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647
> Summary of Memory Usage in PETSc
> [0]Current space PetscMalloc()ed 45936, max space PetscMalloced() 1.13728e+009
> [0]OS cannot compute process memory
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> [0] PetscFOpen(): Opening file Log.0
> [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374784
> [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm
> [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784
> [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> [0] Petsc_DelThreadComm(): Deleting thread communicator data in an MPI_Comm -2080374784
> [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374784
> 
> 
> 
> 2013/10/23 Barry Smith <bsmith at mcs.anl.gov>
> 
>    Don?t run with the -malloc_log Mark did not suggest using that option.
> 
> 
> On Oct 23, 2013, at 8:48 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> 
> > I tried running with the following options :
> > -info
> > -malloc
> > -malloc_log
> > -malloc_dump
> > -malloc_debug
> > -memory_info
> > -log_summary log_summary.txt
> > -log
> > but didn't get a lot wiser however in my case would it be more efficient to keep the ksp solver object, if so should I have one object eg. for the thermal problem and another one for the mechanical?
> > The output from the end of the run:
> >  Total wall clock time actually solving:     856.231474
> > [0] PetscFinalize(): PetscFinalize() called
> > [0] PetscCommDuplicate(): Duplicating a communicator 1140850688 -2080374784 max tags = 2147483647
> > Summary of Memory Usage in PETSc
> > [0]Current space PetscMalloc()ed 45936, max space PetscMalloced() 1.13728e+009
> > [0]OS cannot compute process memory
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374784
> > [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> > [0] PetscGetHostName(): Rejecting domainname, likely is NIS CMP3.
> > [0] PetscFOpen(): Opening file Log.0
> > [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> > [0] Petsc_DelComm_Inner(): Removing reference to PETSc communicator embedded in a user MPI_Comm -2080374784
> > [0] Petsc_DelComm_Outer(): User MPI_Comm 1140850688 is being freed after removing reference from inner PETSc comm to this outer comm
> > [0] PetscCommDestroy(): Deleting PETSc MPI_Comm -2080374784
> > [0] Petsc_DelViewer(): Removing viewer data attribute in an MPI_Comm -2080374784
> > [0] Petsc_DelThreadComm(): Deleting thread communicator data in an MPI_Comm -2080374784
> > [0] Petsc_DelCounter(): Deleting counter data in an MPI_Comm -2080374784
> > [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Object is in wrong state!
> > [0]PETSC ERROR: PetscMallocDumpLog() called without call to PetscMallocSetDumpLog() this is often due to
> >                       setting the option -malloc_log AFTER PetscInitialize() with PetscOptionsInsert() or PetscOptionsInsertFile()!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Wed Oct 23 11:54:03 2013
> > [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
> > [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
> > [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: PetscMallocDumpLog() line 652 in src/sys/memory/D:\PROGRA~1\PETSC-~1.3\src\sys\memory\mtr.c
> > [0]PETSC ERROR: PetscFinalize() line 1185 in src/sys/objects/D:\PROGRA~1\PETSC-~1.3\src\sys\objects\pinit.c
> >
> >
> > 2013/10/19 Mark F. Adams <mfadams at lbl.gov>
> > You can try running a problem that finishes with
> >
> > -malloc
> > -malloc_debug
> > -malloc_dump
> >
> > to see if there are any memory issues.  It looks like you are completely destroying the solver object each time and starting over.  Valgrind would be very useful here because it looks like memory is getting corrupted if the same code runs many times.
> >
> >
> > On Oct 19, 2013, at 3:39 PM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> >
> >> yes, I call the given code many times, what i have seen so far is that changing e.g. the order of destruction statements changes the time of failure,I will try to debug to get more details
> >>
> >>
> >> 2013/10/18 Mark F. Adams <mfadams at lbl.gov>
> >> I would guess there is some memory corruption or leak someplace.  Try running with -on_error_attach_debugger, and try to get some more information.  Do you just repeat a call to the code below many times?  Is this deterministic, does it fail on the same solve each time?
> >>
> >>
> >> On Oct 18, 2013, at 9:30 AM, Einar S?rheim <einar.sorheim at gmail.com> wrote:
> >>
> >>> Uncommenting  KSPSetFromOptions solved my intial problem, however I got a new problem that occurs after many timesteps, in our code we use petsc gamg as a solver for several problems(e.g. thermal and mechanical) at each time step, I have a suspicion it might have something to do with cleanup after solving, the rror appears after many calls to the petsc gamg solver interface, the error message is the following:
> >>>
> >>> [0]PETSC ERROR: ------------------------------------------------------------------------
> >>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> >>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> >>> [0]PETSC ERROR: likely location of problem given in stack below
> >>> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> >>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> >>> [0]PETSC ERROR:       is given.
> >>> [0]PETSC ERROR: [0] MatSetNearNullSpace line 7580 src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c
> >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> >>> [0]PETSC ERROR: Signal received!
> >>> [0]PETSC ERROR: ------------------------------------------------------------------------
> >>> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013
> >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> >>> [0]PETSC ERROR: See docs/index.html for manual pages.
> >>> [0]PETSC ERROR: ------------------------------------------------------------------------
> >>> [0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Fri Oct 18 12:26:04 2013
> >>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib
> >>> [0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013
> >>> [0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/4.1.0.028/em64t" --useThreads=0 --useThreads=0
> >>>
> >>> The source code for the petsc interface:
> >>>
> >>>       subroutine petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond, i_Solver
> >>>      $     ,coords_,i_nodedof)
> >>> #include <finclude/petscsys.h>
> >>> #include <finclude/petscvec.h>
> >>> #include <finclude/petscmat.h>
> >>> #include <finclude/petscpc.h>
> >>> #include <finclude/petscksp.h>
> >>> #include <finclude/petscviewer.h>
> >>>
> >>>       integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof
> >>>       integer i_rowstart_(*),i_columnindex_(*)
> >>>       double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)
> >>>       integer, allocatable  :: i_indices_(:)
> >>> !     Variables:
> >>> !
> >>> !  A       - matrix that defines linear system
> >>> !  ksp    - KSP context
> >>> !  ksp     - KSP context
> >>> !  x, b, u - approx solution, RHS, exact solution vectors
> >>> !
> >>> !      implicit none
> >>>       Vec     x,u,b,vec_coords
> >>>       PC      pc
> >>>       Mat     A,F,Matnull
> >>>       KSP    ksp
> >>>       PetscInt i,j,II,JJ,m,n,i1,imaxiter
> >>>       PetscInt Istart,Iend
> >>>       PetscInt nsteps,one, neg_one
> >>>       PetscErrorCode ierr
> >>>       PetscBool  flg
> >>>       PetscScalar  v,x_array_(1), norm, tol, t0,t1
> >>>       PetscOffset i_x
> >>>       PetscViewer viewer
> >>>
> >>>
> >>>       call PetscTime(t0,ierr)
> >>>       write(*,*) "Enter petsc"
> >>>
> >>>       n      = 1
> >>>       nsteps = 1
> >>>       one    = 1
> >>>       neg_one = 1
> >>>       i1 = 1
> >>>       imaxiter = 1200
> >>>
> >>>
> >>>       call MatCreate(PETSC_COMM_WORLD,A,ierr)
> >>>       call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)
> >>>       call MatSetType(A, MATSEQAIJ,ierr)
> >>>       call MatSetUp(A,ierr)
> >>>       call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)
> >>> !  The matrix is partitioned by contiguous chunks of rows across the
> >>> !  processors.  Determine which rows of the matrix are locally owned.
> >>>
> >>>       call MatGetOwnershipRange(A,Istart,Iend,ierr)
> >>>
> >>>
> >>>       write(*,*) "Start of matrix fill", i_nodes, Istart, Iend
> >>>       do i=1,i_nodes
> >>> !         write(*,*) "Start of loop",i !!, i_indices_(i+1)
> >>> !         i_indices_(i+1) = i
> >>> !         write(*,*) "Start of loop2"
> >>>          do j=i_rowstart_(i),i_rowstart_(i+1)-1
> >>> !            write(*,*) "Start of loop 3"
> >>>             v = r_elements_(j)
> >>> !            write(*,*) i1,i,i1,j-1,v
> >>>             call MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)
> >>>             if (ierr.gt.0) stop
> >>>          end do
> >>>       end do
> >>> !      write(*,*) "End of matrix fill"
> >>>
> >>> !  Assemble matrix, using the 2-step process:
> >>> !       MatAssemblyBegin(), MatAssemblyEnd()
> >>> !  Computations can be done while messages are in transition
> >>> !  by placing code between these two statements.
> >>>
> >>>       call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
> >>>       call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
> >>>
> >>> !  Create parallel vectors.
> >>> !   - When using VecCreate(), the parallel partitioning of the vector
> >>> !     is determined by PETSc at runtime.
> >>> !   - Note: We form 1 vector from scratch and then duplicate as needed.
> >>> !      write(*,*) "Start of vector fill"
> >>>       call VecCreate(PETSC_COMM_WORLD,u,ierr)
> >>>       call VecSetType(u, VECSEQ,ierr)
> >>>       call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)
> >>> !      call VecSetFromOptions(u,ierr)
> >>>       call VecDuplicate(u,b,ierr)
> >>>       call VecDuplicate(b,x,ierr)
> >>>       do i=1,i_nodes
> >>>          call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)
> >>>          call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)
> >>>       enddo
> >>> !  Assemble vector
> >>>
> >>>        call VecAssemblyBegin(x,ierr)
> >>>        call VecAssemblyEnd(x,ierr)
> >>>        call VecAssemblyBegin(b,ierr)
> >>>        call VecAssemblyEnd(b,ierr)
> >>>        call PetscTime(t1,ierr)
> >>> !       write(*,*) "End of vec fill time spent :", t1-t0
> >>> !  Create linear solver context
> >>>
> >>>       call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)
> >>>
> >>>
> >>> !  Set operators. Here the matrix that defines the linear system
> >>> !  also serves as the preconditioning matrix.
> >>>
> >>>       call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
> >>>
> >>>
> >>>       call KSPGetPC(ksp,pc,ierr)
> >>>       tol = 1.e-20
> >>>       call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)
> >>>       select case (i_Precond)
> >>>       case ( 1 )
> >>>          call PCSetType(pc,PCJACOBI,ierr)
> >>>       case ( 2 )
> >>>          call PCSetType(pc,PCILU,ierr)
> >>>          call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);
> >>>          call PCFactorSetUpMatSolverPackage(pc,ierr);
> >>>          call PCFactorGetMatrix(pc,F);
> >>>          call MatSuperluSetILUDropTol(F,1.e-4);
> >>>       case ( 3 )
> >>>          call PCSetType(pc,PCGAMG,ierr)
> >>> !         call PCGAMGInitializePackage()
> >>> !         call PCCreate_GAMG(pc,ierr)
> >>>       case DEFAULT
> >>>          call PCSetType(pc,PCJACOBI,ierr)
> >>>       end select
> >>>
> >>> !      call PCSetType(pc,PCJACOBI,ierr)
> >>>       select case (i_solver)
> >>>       case ( 0 )
> >>>          call KSPSetType(ksp,KSPCG,ierr)
> >>>       case DEFAULT
> >>>          call KSPSetType(ksp,KSPCG,ierr)
> >>>       end select
> >>>       call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)
> >>>       if (i_nodedof==3) then
> >>>          write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof
> >>>          call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)
> >>>          call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)
> >>>          call MatSetNearNullSpace(A,Matnull,ierr)
> >>>          call MatNullSpaceDestroy(Matnull,ierr)
> >>>
> >>>          write(*,*) "Finish setting null space ierr =", ierr
> >>> !         call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )
> >>>
> >>>       end if
> >>> !      call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL, PETSC_NULL,ierr)
> >>> !  Set runtime options, e.g.,
> >>> !      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
> >>> !  These options will override those specified above as long as
> >>> !  KSPSetFromOptions() is called _after_ any other customization
> >>> !  routines.
> >>>
> >>>       call KSPSetFromOptions(ksp,ierr)
> >>>       call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)
> >>>       call KSPSetUp(ksp,ierr)
> >>>       call PetscTime(t0,ierr)
> >>>       write(*,*) "Time setup", t0-t1
> >>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >>> !                      Solve the linear system
> >>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >>>
> >>>       call KSPSolve(ksp,b,x,ierr)
> >>>       call PetscTime(t1,ierr)
> >>>       write(*,*) "end solve, time used:",t1-t0
> >>>       call KSPGetSolution(ksp,x,ierr)
> >>>       call VecGetArray(x,x_array_,i_x,ierr)
> >>>       do i=1,i_nodes
> >>>          r_x_(i) = x_array_(i_x + i)
> >>>       enddo
> >>>       call VecRestoreArray(x,x_array_,i_x,ierr)
> >>>
> >>>
> >>> !  Free work space.  All PETSc objects should be destroyed when they
> >>> !  are no longer needed.
> >>> !      call PCDestroy(pc,ierr)
> >>>
> >>>       call VecDestroy(u,ierr)
> >>>       call VecDestroy(x,ierr)
> >>>       call VecDestroy(b,ierr)
> >>>       call MatDestroy(A,ierr)
> >>>       if (i_nodedof==3) then
> >>>          call VecDestroy(vec_coords,ierr)
> >>>          call MatDestroy(Matnull,ierr)
> >>>       endif
> >>>       call KSPDestroy(ksp,ierr)
> >>>       call PetscTime(t0,ierr)
> >>>       write(*,*) "time celan up :", t0-t1
> >>>
> >>>       end
> >>>
> >>> --
> >>> Einar S?rheim
> >>> <Mail Attachment>
> >>
> >>
> >>
> >>
> >> --
> >> Einar S?rheim
> >
> >
> >
> >
> > --
> > Einar S?rheim
> 
> 
> 
> 
> -- 
> Einar S?rheim
> <log_summary.txt>


From cedric.doucet at inria.fr  Thu Oct 24 09:29:46 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Thu, 24 Oct 2013 16:29:46 +0200 (CEST)
Subject: [petsc-users] Partitioning meshes with DMPlex
In-Reply-To: <CAMYG4GmVtj1EXn_cAdd43wwB0v24p1_qp3zQV5v3iOYrecu0yQ@mail.gmail.com>
Message-ID: <983188657.61824.1382624986818.JavaMail.root@inria.fr>

Thank you Matt and Jed for your answers! 
As far as I understand, DMPlexDistribute call DMPlexCreatePartition and is exactly what I am looking for. 

C?dric 

----- Mail original -----

> De: "Matthew Knepley" <knepley at gmail.com>
> ?: "Jed Brown" <jedbrown at mcs.anl.gov>
> Cc: "Cedric Doucet" <cedric.doucet at inria.fr>, petsc-users at mcs.anl.gov
> Envoy?: Jeudi 24 Octobre 2013 00:25:40
> Objet: Re: [petsc-users] Partitioning meshes with DMPlex

> On Wed, Oct 23, 2013 at 10:43 AM, Jed Brown < jedbrown at mcs.anl.gov >
> wrote:

> > Cedric Doucet < cedric.doucet at inria.fr > writes:
> 

> > > Hello,
> 
> > >
> 
> > > what is the best way to partition unstructured meshes stored with
> > > DMPlex?
> 

> > You can use DMPlexCreatePartition, which supports metis and chaco.
> 

> > Matt, DMPlexCreatePartition documentation has left out the
> > partitioner "name".
> 
> I assume what you really want is DMPlexDistribute().

> Matt

> > > Does one have to create an adjacency matrix and use
> > > MatPartitioning
> > > routines?
> 
> > > I do not want to partition the mesh graph (by cutting edges) but
> > > having a true domain decomposition (sets of cells per processor).
> 
> > >
> 
> > > Thank you very much for your help!
> 
> > >
> 
> > > Best regards,
> 
> > >
> 
> > > C?dric Doucet
> 

> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131024/a768231d/attachment.html>

From lu_qin_2000 at yahoo.com  Thu Oct 24 10:33:43 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 24 Oct 2013 08:33:43 -0700 (PDT)
Subject: [petsc-users] Could not find make when building PETSc in Windows 7
Message-ID: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>

Hello,

I am trying to build PETSc in Windows 7 with following steps:

1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
2. Added C:\cygwin64\bin to Windows environmental variable Path.
3. Downloaded and unzipped PETSc 3.4.2.
4. Run mintty.exe to open a cygwin terminal.
5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'

However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.

What should I do to let configure know the path of make?
?
Many thanks for your help,
?
Regard,
Qin?

From jedbrown at mcs.anl.gov  Thu Oct 24 10:36:08 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 24 Oct 2013 10:36:08 -0500
Subject: [petsc-users] Could not find make when building PETSc in
	Windows 7
In-Reply-To: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
Message-ID: <87a9hybs0n.fsf@mcs.anl.gov>

Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Hello,
>
> I am trying to build PETSc in Windows 7 with following steps:
>
> 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> 3. Downloaded and unzipped PETSc 3.4.2.
> 4. Run mintty.exe to open a cygwin terminal.
> 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'
>
> However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.

You need to pass the full path to the make executable (or add it to your
path), or you can use --download-make.

> What should I do to let configure know the path of make?
> ?
> Many thanks for your help,
> ?
> Regard,
> Qin?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131024/c766e734/attachment.pgp>

From lu_qin_2000 at yahoo.com  Thu Oct 24 10:43:28 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 24 Oct 2013 08:43:28 -0700 (PDT)
Subject: [petsc-users] Could not find make when building PETSc in
	Windows 7
In-Reply-To: <87a9hybs0n.fsf@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
Message-ID: <1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>

Jed,
?
As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make in confugure, it gave the same error.
?
Thanks,
Qin



On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Hello,
>
> I am trying to build PETSc in Windows 7 with following steps:
>
> 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> 3. Downloaded and unzipped PETSc 3.4.2.
> 4. Run mintty.exe to open a cygwin terminal.
> 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'
>
> However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.

You need to pass the full path to the make executable (or add it to your
path), or you can use --download-make.


> What should I do to let configure know the path of make?
> ?
> Many thanks for your help,
> ?
> Regard,
> Qin?   

From balay at mcs.anl.gov  Thu Oct 24 10:46:22 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 24 Oct 2013 10:46:22 -0500 (CDT)
Subject: [petsc-users] Could not find make when building PETSc
 in	Windows 7
In-Reply-To: <1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310241044350.19655@mcs.anl.gov>

On Thu, 24 Oct 2013, Qin Lu wrote:

> Jed,
> ?
> As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, 

You shouldn't have to do this. Perhaps cygwin is not setup properly.

Also cygwin notation is /usr/bin - not C:\cygwin64\bin

Are you running the commands in 'mintty' terminal or 'cmd' terminal?

> and also tried to run ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make in confugure, it gave the same error.

--download-make is not in petsc-3.4

BTW: the latest patch update is petsc-3.4.3

Satish


> ?
> Thanks,
> Qin
> 
> 
> 
> On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Qin Lu <lu_qin_2000 at yahoo.com> writes:
> 
> > Hello,
> >
> > I am trying to build PETSc in Windows 7 with following steps:
> >
> > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > 3. Downloaded and unzipped PETSc 3.4.2.
> > 4. Run mintty.exe to open a cygwin terminal.
> > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'
> >
> > However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> 
> You need to pass the full path to the make executable (or add it to your
> path), or you can use --download-make.
> 
> 
> > What should I do to let configure know the path of make?
> > ?
> > Many thanks for your help,
> > ?
> > Regard,
> > Qin?
> 

From balay at mcs.anl.gov  Thu Oct 24 10:47:53 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 24 Oct 2013 10:47:53 -0500 (CDT)
Subject: [petsc-users] Could not find make when building PETSc
 in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310241044350.19655@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241044350.19655@mcs.anl.gov>
Message-ID: <alpine.LFD.2.03.1310241047020.19655@mcs.anl.gov>

Also what do you have for:

echo $PATH
which make

Retry wiht petsc-3.4.3 - and if you still have issues - send us configure.log

Satish

On Thu, 24 Oct 2013, Satish Balay wrote:

> On Thu, 24 Oct 2013, Qin Lu wrote:
> 
> > Jed,
> > ?
> > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, 
> 
> You shouldn't have to do this. Perhaps cygwin is not setup properly.
> 
> Also cygwin notation is /usr/bin - not C:\cygwin64\bin
> 
> Are you running the commands in 'mintty' terminal or 'cmd' terminal?
> 
> > and also tried to run ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make in confugure, it gave the same error.
> 
> --download-make is not in petsc-3.4
> 
> BTW: the latest patch update is petsc-3.4.3
> 
> Satish
> 
> 
> > ?
> > Thanks,
> > Qin
> > 
> > 
> > 
> > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > 
> > > Hello,
> > >
> > > I am trying to build PETSc in Windows 7 with following steps:
> > >
> > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > 4. Run mintty.exe to open a cygwin terminal.
> > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'
> > >
> > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > 
> > You need to pass the full path to the make executable (or add it to your
> > path), or you can use --download-make.
> > 
> > 
> > > What should I do to let configure know the path of make?
> > > ?
> > > Many thanks for your help,
> > > ?
> > > Regard,
> > > Qin?
> > 
> 

From lu_qin_2000 at yahoo.com  Thu Oct 24 11:26:44 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 24 Oct 2013 09:26:44 -0700 (PDT)
Subject: [petsc-users] Could not find make when building PETSc
	in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310241047020.19655@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241044350.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241047020.19655@mcs.anl.gov>
Message-ID: <1382632004.15624.YahooMailNeo@web160203.mail.bf1.yahoo.com>

Satish,

I ran the configure in mintty's terminal.
?
"echo $PATH" showed PATH included /usr/bin. "which make" showed "/usr/bin/make". I tried petsc 3.4.3 with either --with-make or --download-make option and configure got the same error.
?
The configure.log is attached.
?
Is there simple anyway to test if cygwin?has been set up properly?
?
Thanks,
Qin?


On Thursday, October 24, 2013 10:47 AM, Satish Balay <balay at mcs.anl.gov> wrote:
Also what do you have for:

echo $PATH
which make

Retry wiht petsc-3.4.3 - and if you still have issues - send us configure.log

Satish


On Thu, 24 Oct 2013, Satish Balay wrote:

> On Thu, 24 Oct 2013, Qin Lu wrote:
> 
> > Jed,
> > ?
> > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, 
> 
> You shouldn't have to do this. Perhaps cygwin is not setup properly.
> 
> Also cygwin notation is /usr/bin - not C:\cygwin64\bin
> 
> Are you running the commands in 'mintty' terminal or 'cmd' terminal?
> 
> > and also tried to run ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make in confugure, it gave the same error.
> 
> --download-make is not in petsc-3.4
> 
> BTW: the latest patch update is petsc-3.4.3
> 
> Satish
> 
> 
> > ?
> > Thanks,
> > Qin
> > 
> > 
> > 
> > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > 
> > > Hello,
> > >
> > > I am trying to build PETSc in Windows 7 with following steps:
> > >
> > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > 4. Run mintty.exe to open a cygwin terminal.
> > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl'
> > >
> > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > 
> > You need to pass the full path to the make executable (or add it to your
> > path), or you can use --download-make.
> > 
> > 
> > > What should I do to let configure know the path of make?
> > > ?
> > > Many thanks for your help,
> > > ?
> > > Regard,
> > > Qin?
> > 
> ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 27419 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131024/96cd4b89/attachment-0001.obj>

From chetan.jhurani at gmail.com  Thu Oct 24 11:29:00 2013
From: chetan.jhurani at gmail.com (Chetan Jhurani)
Date: Thu, 24 Oct 2013 09:29:00 -0700
Subject: [petsc-users] Could not find make when building PETSc
	in	Windows 7
In-Reply-To: <1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
Message-ID: <52694aca.ea42420a.1596.5523@mx.google.com>

I vaguely recall seeing the "could not locate make" error
when a non-cygwin python is picked up first in path.
So ensure that "which python" points to cygwin python.

Another issue is that the --with-make argument, if it is
needed, should be /cygdrive/C/cygwin64/bin (or with some
other drive letter in between where cygwin is installed).

Chetan

> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> Sent: Thursday, October 24, 2013 8:43 AM
> To: Jed Brown; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> 
> Jed,
> 
> As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> in confugure, it gave the same error.
> 
> Thanks,
> Qin
> 
> 
> 
> On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Qin Lu <lu_qin_2000 at yahoo.com> writes:
> 
> > Hello,
> >
> > I am trying to build PETSc in Windows 7 with following steps:
> >
> > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > 3. Downloaded and unzipped PETSc 3.4.2.
> > 4. Run mintty.exe to open a cygwin terminal.
> > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> fc='win32fe ifort' --with-cxx='win32fe icl'
> >
> > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> 
> You need to pass the full path to the make executable (or add it to your
> path), or you can use --download-make.
> 
> 
> > What should I do to let configure know the path of make?
> >
> > Many thanks for your help,
> >
> > Regard,
> > Qin


From lu_qin_2000 at yahoo.com  Thu Oct 24 11:39:58 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 24 Oct 2013 09:39:58 -0700 (PDT)
Subject: [petsc-users] Could not find make when building PETSc
	in	Windows 7
In-Reply-To: <52694aca.ea42420a.1596.5523@mx.google.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
Message-ID: <1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>

Chetan

Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.

Thanks a lot for all you guys'?helps.

Qin?

On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
I vaguely recall seeing the "could not locate make" error
when a non-cygwin python is picked up first in path.
So ensure that "which python" points to cygwin python.

Another issue is that the --with-make argument, if it is
needed, should be /cygdrive/C/cygwin64/bin (or with some
other drive letter in between where cygwin is installed).

Chetan


> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> Sent: Thursday, October 24, 2013 8:43 AM
> To: Jed Brown; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> 
> Jed,
> 
> As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> in confugure, it gave the same error.
> 
> Thanks,
> Qin
> 
> 
> 
> On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Qin Lu <lu_qin_2000 at yahoo.com> writes:
> 
> > Hello,
> >
> > I am trying to build PETSc in Windows 7 with following steps:
> >
> > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > 3. Downloaded and unzipped PETSc 3.4.2.
> > 4. Run mintty.exe to open a cygwin terminal.
> > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> fc='win32fe ifort' --with-cxx='win32fe icl'
> >
> > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> 
> You need to pass the full path to the make executable (or add it to your
> path), or you can use --download-make.
> 
> 
> > What should I do to let configure know the path of make?
> >
> > Many thanks for your help,
> >
> > Regard,
> > Qin

From balay at mcs.anl.gov  Thu Oct 24 12:38:14 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 24 Oct 2013 12:38:14 -0500 (CDT)
Subject: [petsc-users] Could not find make when building
 PETSc	in	Windows 7
In-Reply-To: <1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>

Perhaps we should add back the windows-python check?

[since we haven't yet added the windows python support]

Satish

On Thu, 24 Oct 2013, Qin Lu wrote:

> Chetan
> 
> Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> 
> Thanks a lot for all you guys'?helps.
> 
> Qin?
> 
> On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> I vaguely recall seeing the "could not locate make" error
> when a non-cygwin python is picked up first in path.
> So ensure that "which python" points to cygwin python.
> 
> Another issue is that the --with-make argument, if it is
> needed, should be /cygdrive/C/cygwin64/bin (or with some
> other drive letter in between where cygwin is installed).
> 
> Chetan
> 
> 
> > -----Original Message-----
> > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > Sent: Thursday, October 24, 2013 8:43 AM
> > To: Jed Brown; petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > 
> > Jed,
> > 
> > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > in confugure, it gave the same error.
> > 
> > Thanks,
> > Qin
> > 
> > 
> > 
> > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > 
> > > Hello,
> > >
> > > I am trying to build PETSc in Windows 7 with following steps:
> > >
> > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > 4. Run mintty.exe to open a cygwin terminal.
> > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > fc='win32fe ifort' --with-cxx='win32fe icl'
> > >
> > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > 
> > You need to pass the full path to the make executable (or add it to your
> > path), or you can use --download-make.
> > 
> > 
> > > What should I do to let configure know the path of make?
> > >
> > > Many thanks for your help,
> > >
> > > Regard,
> > > Qin
> 

From balay at mcs.anl.gov  Thu Oct 24 14:05:17 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 24 Oct 2013 14:05:17 -0500 (CDT)
Subject: [petsc-users] Could not find make when building
 PETSc	in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
Message-ID: <alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>

pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.

It will go into 'maint' later..

Satish

On Thu, 24 Oct 2013, Satish Balay wrote:

> Perhaps we should add back the windows-python check?
> 
> [since we haven't yet added the windows python support]
> 
> Satish
> 
> On Thu, 24 Oct 2013, Qin Lu wrote:
> 
> > Chetan
> > 
> > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > 
> > Thanks a lot for all you guys'?helps.
> > 
> > Qin?
> > 
> > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > I vaguely recall seeing the "could not locate make" error
> > when a non-cygwin python is picked up first in path.
> > So ensure that "which python" points to cygwin python.
> > 
> > Another issue is that the --with-make argument, if it is
> > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > other drive letter in between where cygwin is installed).
> > 
> > Chetan
> > 
> > 
> > > -----Original Message-----
> > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > Sent: Thursday, October 24, 2013 8:43 AM
> > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > 
> > > Jed,
> > > 
> > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > in confugure, it gave the same error.
> > > 
> > > Thanks,
> > > Qin
> > > 
> > > 
> > > 
> > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > 
> > > > Hello,
> > > >
> > > > I am trying to build PETSc in Windows 7 with following steps:
> > > >
> > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > >
> > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > 
> > > You need to pass the full path to the make executable (or add it to your
> > > path), or you can use --download-make.
> > > 
> > > 
> > > > What should I do to let configure know the path of make?
> > > >
> > > > Many thanks for your help,
> > > >
> > > > Regard,
> > > > Qin
> > 
> 

From christophe.ortiz at ciemat.es  Fri Oct 25 04:47:21 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Fri, 25 Oct 2013 11:47:21 +0200
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
Message-ID: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>

Hi guys,

I was having some troubles with my code (dof >> 1) and I decided to go back
to simple things. Therefore, I took ex25.c, which is a nice case for me:
diffusion of 2 species and 1 reaction.

First, to have a better mesh I set 256 points (instead of the 11) in

  ierr = DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,256,2,2,NULL,&da);

Then, I did something simple. I removed reaction terms by commenting

ierr = TSSetRHSFunction(ts,NULL,FormRHSFunction,&user);CHKERRQ(ierr);

So we are left with 2 diffusion equations:

u_t -alpha u_xx = 0
v_t - alpha v_xx = 0

Now, do something simple:
In FormInitialSolution, for variable v, change it and simply set it to 5
(for instance), ie, v will have uniform values from i=0 to i=mx-1.

Now, for boundary conditions (Dirichlet here), set vleft and vright to 10,
for instance.

Since values at the boundaries (10.0) are higher than values of initial
solution (5.0), what you expect is some diffusion towards the center from
the boundaries. Since we use Dirichlet boundary conditions here, you expect
the values of v at i=0 and i=mx-1 to be 10.0. It can't be something else.
Well, that's not the case. I see that the value of v at i=0 and at i=mx-1
is always the initial value (t=0). Here is a sample of the final values
after diffusion:

0    5.000000
1    8.951153
2    8.414078
3    7.964193
4    7.581196
...
...
77    5.000235
78    5.000207
79    5.000182
80    5.000161
...
...
252    7.964193
253    8.414078
254    8.951153
255    5.000000

Then I checked the value of v inside IFunction at each timestep. I found
out that each time IFunction is called, the first value of v that is used
to evaluate F is the initial value (5.0), instead of of the value set by
Dirichlet (10). For next evaluations within the same timestep, it is 10.0,
the value imposed by Dirichlet boundary conditions.
Some output:

      TSAdapt 'basic': step 324 accepted t=9.73302    + 9.496e-02
wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.569e-02
5.000000
10.000000
10.000000
10.000000
10.000000

      TSAdapt 'basic': step 325 accepted t=9.82798    + 9.569e-02
wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.642e-02
5.000000
10.000000
10.000000
10.000000
10.000000

Is it a bug or did I miss something ? I would expect that to evaluate F
within IFunction, it would always use the value set by Dirichlet.


I use petsc 3.4.1and configured it with
 --with-mpi=0 --with-fc=ifort --with-cc=icc --with-debugging=1
--with-scalar-type=real --with-precision=double
--with-blas-lapack-dir=/opt/intel/mkl.

The options I used to run the example are
-ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
-ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
-snes_linesearch_type bt.


Christophe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/290e9db3/attachment.html>

From christophe.ortiz at ciemat.es  Fri Oct 25 06:22:55 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Fri, 25 Oct 2013 13:22:55 +0200
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
Message-ID: <CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>

After playing around, I found something interesting, an option that I have
never seen in any example: TSSetEquationType. I did a TSGetEquationType and
I got TS_EQ_UNSPECIFIED. Then I thought, ok, let's try to specify the
equation type and I added the following line to the code

  ierr = TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1);

Well...the results is totally different, and in my opinion much more
consistent. The value at the boundary is now the expected one:

0    10.000000     <-----
1    9.888647
2    9.777382
3    9.666293
4    9.555467

...
...
112    5.011530
113    5.010665
114    5.009880
115    5.009169
...
...
252    9.666293
253    9.777382
254    9.888647
255    10.000000     <----


Moreover, I see much more effect of diffusion. Without this option,
diffusion is always very weak, whatever the diffusion coefficient one puts,
which always surprised me.

What is the effect of the TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1); ?

Christophe


On Fri, Oct 25, 2013 at 11:47 AM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

> Hi guys,
>
> I was having some troubles with my code (dof >> 1) and I decided to go
> back to simple things. Therefore, I took ex25.c, which is a nice case for
> me: diffusion of 2 species and 1 reaction.
>
> First, to have a better mesh I set 256 points (instead of the 11) in
>
>   ierr =
> DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,256,2,2,NULL,&da);
>
> Then, I did something simple. I removed reaction terms by commenting
>
> ierr = TSSetRHSFunction(ts,NULL,FormRHSFunction,&user);CHKERRQ(ierr);
>
> So we are left with 2 diffusion equations:
>
> u_t -alpha u_xx = 0
> v_t - alpha v_xx = 0
>
> Now, do something simple:
> In FormInitialSolution, for variable v, change it and simply set it to 5
> (for instance), ie, v will have uniform values from i=0 to i=mx-1.
>
> Now, for boundary conditions (Dirichlet here), set vleft and vright to 10,
> for instance.
>
> Since values at the boundaries (10.0) are higher than values of initial
> solution (5.0), what you expect is some diffusion towards the center from
> the boundaries. Since we use Dirichlet boundary conditions here, you expect
> the values of v at i=0 and i=mx-1 to be 10.0. It can't be something else.
> Well, that's not the case. I see that the value of v at i=0 and at i=mx-1
> is always the initial value (t=0). Here is a sample of the final values
> after diffusion:
>
> 0    5.000000
> 1    8.951153
> 2    8.414078
> 3    7.964193
> 4    7.581196
> ...
> ...
> 77    5.000235
> 78    5.000207
> 79    5.000182
> 80    5.000161
> ...
> ...
> 252    7.964193
> 253    8.414078
> 254    8.951153
> 255    5.000000
>
> Then I checked the value of v inside IFunction at each timestep. I found
> out that each time IFunction is called, the first value of v that is used
> to evaluate F is the initial value (5.0), instead of of the value set by
> Dirichlet (10). For next evaluations within the same timestep, it is 10.0,
> the value imposed by Dirichlet boundary conditions.
> Some output:
>
>       TSAdapt 'basic': step 324 accepted t=9.73302    + 9.496e-02
> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.569e-02
> 5.000000
> 10.000000
> 10.000000
> 10.000000
> 10.000000
>
>       TSAdapt 'basic': step 325 accepted t=9.82798    + 9.569e-02
> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.642e-02
> 5.000000
> 10.000000
> 10.000000
> 10.000000
> 10.000000
>
> Is it a bug or did I miss something ? I would expect that to evaluate F
> within IFunction, it would always use the value set by Dirichlet.
>
>
> I use petsc 3.4.1and configured it with
>  --with-mpi=0 --with-fc=ifort --with-cc=icc --with-debugging=1
> --with-scalar-type=real --with-precision=double
> --with-blas-lapack-dir=/opt/intel/mkl.
>
> The options I used to run the example are
> -ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
> -ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
> -snes_linesearch_type bt.
>
>
> Christophe
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/2881b446/attachment.html>

From cedric.doucet at inria.fr  Fri Oct 25 07:23:03 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Fri, 25 Oct 2013 14:23:03 +0200 (CEST)
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <983188657.61824.1382624986818.JavaMail.root@inria.fr>
Message-ID: <1647503755.70929.1382703783302.JavaMail.root@inria.fr>

Hello, 

I've noticed that DMPlexCreateFromCellList assumes that cells have the same number of vertices (numcorners argument). 
What should be done when one wants to create a DMPlex from a mesh containing different types of cells? 
Does one have to create several DMPlex structures and merge them? 
Does one have to create a unique DMPlex by hand? 

Thank you very much for your help. 

C?dric 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/0dc98d7f/attachment-0001.html>

From knepley at gmail.com  Fri Oct 25 09:31:59 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Oct 2013 09:31:59 -0500
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <1647503755.70929.1382703783302.JavaMail.root@inria.fr>
References: <983188657.61824.1382624986818.JavaMail.root@inria.fr>
	<1647503755.70929.1382703783302.JavaMail.root@inria.fr>
Message-ID: <CAMYG4G=GdmQAHBJAiDCM=_A_4y9zYtEg_ou-zrY6M=oHPmFebg@mail.gmail.com>

On Fri, Oct 25, 2013 at 7:23 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:

> Hello,
>
> I've noticed that DMPlexCreateFromCellList assumes that cells have the
> same number of vertices (numcorners argument).
> What should be done when one wants to create a DMPlex from a mesh
> containing different types of cells?
> Does one have to create several DMPlex structures and merge them?
> Does one have to create a unique DMPlex by hand?
>

The code is very short:

  ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
  for (c = 0; c < numCells; ++c) {
    ierr = DMPlexSetConeSize(dm, c, numCorners);CHKERRQ(ierr);
  }
  ierr = DMSetUp(dm);CHKERRQ(ierr);
  ierr = DMGetWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr);
  for (c = 0; c < numCells; ++c) {
    for (p = 0; p < numCorners; ++p) {
      cone[p] = cells[c*numCorners+p]+numCells;
    }
    ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
  }
  ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr);
  ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
  ierr = DMPlexStratify(dm);CHKERRQ(ierr);

This code is all in plexcreate.c. If you want different cells, you can
change numCorners for each cell.
I could make a convenience form if necessary.

   Matt


>
> Thank you very much for your help.
>
> C?dric
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/e2c480f0/attachment.html>

From torquil at gmail.com  Fri Oct 25 11:33:56 2013
From: torquil at gmail.com (=?UTF-8?Q?Torquil_Macdonald_S=C3=B8rensen?=)
Date: Fri, 25 Oct 2013 18:33:56 +0200
Subject: [petsc-users] KSPSolve convergence and norms
Message-ID: <CAD6PAKN_5WjWdQfonHLut9gSxD1b6DrToO5P+i5GW7=4q=V+bA@mail.gmail.com>

Hi!

I'm using PETSc 3.4.3, with only default runtime settings, to solve Ax=b,
for complex numbers (I have used --with-scalar-type=complex).

After reading sections 4.3.1, 4.3.2, 4.3.3 in the manual, I decided to
investigate the convergence in my program, since I'm having some problems
when the system is larger. So, I'm running it with the options

-ksp_monitor_true_residual -ksp_converged_reason -ksp_view

The output shows that the relative tolerance required for convergence is
1e-5, but none of the reported norms are that small. Despite this,
convergence is reported (CONVERGED_RTOL).

I though that CONVERGED_RTOL should indicate that ||r||_2/||b||_2 < 1e-5,
where r is the preconditioned residual?

Here is the output of my program (which includes some code to compute the
l_2 norm of b, just before KSPSolve is run):

*************************************************************
Norm of b: 0.179829
  0 KSP preconditioned resid norm 2.508789303280e+04 true resid norm
1.798290175843e-01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.856991132478e+00 true resid norm
3.186462510386e-01 ||r(i)||/||b|| 1.771940120227e+00
  2 KSP preconditioned resid norm 1.704678606994e-01 true resid norm
4.004071321891e-02 ||r(i)||/||b|| 2.226599119363e-01
Linear solve converged due to CONVERGED_RTOL iterations 2
KSP Object: 1 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: ilu
    ILU: out-of-place factorization
    0 levels of fill
    tolerance for zero pivot 2.22045e-14
    using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
    matrix ordering: natural
    factor fill ratio given 1, needed 1
      Factored matrix follows:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=18, cols=18
          package used to perform factorization: petsc
          total: nonzeros=144, allocated nonzeros=144
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 9 nodes, limit used is 5
  linear system matrix = precond matrix:
  Matrix Object:   1 MPI processes
    type: seqaij
    rows=18, cols=18
    total: nonzeros=144, allocated nonzeros=360
    total number of mallocs used during MatSetValues calls =18
      using I-node routines: found 9 nodes, limit used is 5
************************************************************************************************

Here is the code I used to get the l_2 norm of b:

PetscReal val;
VecNorm(b, NORM_2, &val);
PetscPrintf(PETSC_COMM_WORLD, "Norm of b: %G\n", val);

Best regards
Torquil S?rensen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/3ac04d74/attachment.html>

From knepley at gmail.com  Fri Oct 25 11:59:45 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Oct 2013 11:59:45 -0500
Subject: [petsc-users] KSPSolve convergence and norms
In-Reply-To: <CAD6PAKN_5WjWdQfonHLut9gSxD1b6DrToO5P+i5GW7=4q=V+bA@mail.gmail.com>
References: <CAD6PAKN_5WjWdQfonHLut9gSxD1b6DrToO5P+i5GW7=4q=V+bA@mail.gmail.com>
Message-ID: <CAMYG4GntLCk9WCOE84tfMbPFG=xGc03DY5oZvWShvOm1grAuwQ@mail.gmail.com>

On Fri, Oct 25, 2013 at 11:33 AM, Torquil Macdonald S?rensen <
torquil at gmail.com> wrote:

> Hi!
>
> I'm using PETSc 3.4.3, with only default runtime settings, to solve Ax=b,
> for complex numbers (I have used --with-scalar-type=complex).
>
> After reading sections 4.3.1, 4.3.2, 4.3.3 in the manual, I decided to
> investigate the convergence in my program, since I'm having some problems
> when the system is larger. So, I'm running it with the options
>
> -ksp_monitor_true_residual -ksp_converged_reason -ksp_view
>
> The output shows that the relative tolerance required for convergence is
> 1e-5, but none of the reported norms are that small. Despite this,
> convergence is reported (CONVERGED_RTOL).
>
> I though that CONVERGED_RTOL should indicate that ||r||_2/||b||_2 < 1e-5,
> where r is the preconditioned residual?
>
> Here is the output of my program (which includes some code to compute the
> l_2 norm of b, just before KSPSolve is run):
>
> *************************************************************
> Norm of b: 0.179829
>   0 KSP preconditioned resid norm 2.508789303280e+04 true resid norm
> 1.798290175843e-01 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.856991132478e+00 true resid norm
> 3.186462510386e-01 ||r(i)||/||b|| 1.771940120227e+00
>   2 KSP preconditioned resid norm 1.704678606994e-01 true resid norm
> 4.004071321891e-02 ||r(i)||/||b|| 2.226599119363e-01
>

So the relative preconditioned residual is:

  1.7e-01 / 2.5e+04 < 1.0e-5

while the relative true residual is 2.2e-1. This happens because you are
using ILU(0). It can be really
crappy, and in this case the preconditioner is very ill-conditioned. I
can't imagine a problem where I
would recommend using ILU, but its our default because its the only
black-box PC that exists.

  Thanks

     Matt


> Linear solve converged due to CONVERGED_RTOL iterations 2
> KSP Object: 1 MPI processes
>   type: gmres
>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: ilu
>     ILU: out-of-place factorization
>     0 levels of fill
>     tolerance for zero pivot 2.22045e-14
>     using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>     matrix ordering: natural
>     factor fill ratio given 1, needed 1
>       Factored matrix follows:
>         Matrix Object:         1 MPI processes
>           type: seqaij
>           rows=18, cols=18
>           package used to perform factorization: petsc
>           total: nonzeros=144, allocated nonzeros=144
>           total number of mallocs used during MatSetValues calls =0
>             using I-node routines: found 9 nodes, limit used is 5
>   linear system matrix = precond matrix:
>   Matrix Object:   1 MPI processes
>     type: seqaij
>     rows=18, cols=18
>     total: nonzeros=144, allocated nonzeros=360
>     total number of mallocs used during MatSetValues calls =18
>       using I-node routines: found 9 nodes, limit used is 5
>
> ************************************************************************************************
>
> Here is the code I used to get the l_2 norm of b:
>
> PetscReal val;
> VecNorm(b, NORM_2, &val);
> PetscPrintf(PETSC_COMM_WORLD, "Norm of b: %G\n", val);
>
> Best regards
> Torquil S?rensen
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/edcfb9c8/attachment.html>

From bisheshkh at gmail.com  Fri Oct 25 12:09:32 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 25 Oct 2013 19:09:32 +0200
Subject: [petsc-users] using petsc tools to solve isolated irregular domains
	with finite difference
Message-ID: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>

Dear all,
I would like to know if some of the petsc objects that I have not used so
far (IS, DMPlex, PetscSection) could be useful in the following case (of
irregular domains):

Let's say that I have a 3D binary image (a cube).
The binary information of the image partitions the cube into a
computational domain and non-computational domain.
I must solve a pde (say a Poisson equation) only on the computational
domains (e.g: two isolated spheres within the cube). I'm using finite
difference and say a dirichlet boundary condition

I know that I can create a dmda that will let me access the information
from this 3D binary image, get all the coefficients, rhs values etc using
the natural indexing (i,j,k).

Now, I would like to create a matrix corresponding to the laplace operator
(e.g. with standard 7 pt. stencil), and the corresponding RHS that takes
care of the dirchlet values too.
But in this matrix it should have the rows corresponding to the nodes only
on the computational domain. It would be nice if I can easily (using
(i,j,k) indexing) put on the rhs dirichlet values corresponding to the
boundary points.
Then, once the system is solved, put the values of the solution back to the
corresponding positions in the binary image.
Later, I might have to extend this for the staggered grid case too.
So is petscsection or dmplex suitable for this so that I can set up the
matrix with something like DMCreateMatrix ? Or what would you suggest as a
suitable approach to this problem ?

I have looked at the manual and that led me to search for a simpler
examples in petsc src directories. But most of the ones I encountered are
with FEM (and I'm not familiar at all with FEM, so these examples serve
more as a distraction with FEM jargon!)

Thanks,
Bishesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/88f1ba9b/attachment.html>

From knepley at gmail.com  Fri Oct 25 13:18:05 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Oct 2013 13:18:05 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
Message-ID: <CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>

On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:

> Dear all,
> I would like to know if some of the petsc objects that I have not used so
> far (IS, DMPlex, PetscSection) could be useful in the following case (of
> irregular domains):
>
> Let's say that I have a 3D binary image (a cube).
> The binary information of the image partitions the cube into a
> computational domain and non-computational domain.
> I must solve a pde (say a Poisson equation) only on the computational
> domains (e.g: two isolated spheres within the cube). I'm using finite
> difference and say a dirichlet boundary condition
>
> I know that I can create a dmda that will let me access the information
> from this 3D binary image, get all the coefficients, rhs values etc using
> the natural indexing (i,j,k).
>
> Now, I would like to create a matrix corresponding to the laplace operator
> (e.g. with standard 7 pt. stencil), and the corresponding RHS that takes
> care of the dirchlet values too.
> But in this matrix it should have the rows corresponding to the nodes only
> on the computational domain. It would be nice if I can easily (using
> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
> boundary points.
> Then, once the system is solved, put the values of the solution back to
> the corresponding positions in the binary image.
> Later, I might have to extend this for the staggered grid case too.
> So is petscsection or dmplex suitable for this so that I can set up the
> matrix with something like DMCreateMatrix ? Or what would you suggest as a
> suitable approach to this problem ?
>
> I have looked at the manual and that led me to search for a simpler
> examples in petsc src directories. But most of the ones I encountered are
> with FEM (and I'm not familiar at all with FEM, so these examples serve
> more as a distraction with FEM jargon!)
>

It sounds like the right solution for this is to use PetscSection on top of
DMDA. I am working on this, but it is really
alpha code. If you feel comfortable with that level of development, we can
help you. If not, just put the identity into
the rows you do not use on the full cube. It will not hurt scalability or
convergence.

  Matt


> Thanks,
> Bishesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/dd431835/attachment.html>

From torquil at gmail.com  Fri Oct 25 13:18:23 2013
From: torquil at gmail.com (=?ISO-8859-1?Q?Torquil_Macdonald_S=F8rensen?=)
Date: Fri, 25 Oct 2013 20:18:23 +0200
Subject: [petsc-users] KSPSolve convergence and norms
In-Reply-To: <CAMYG4GntLCk9WCOE84tfMbPFG=xGc03DY5oZvWShvOm1grAuwQ@mail.gmail.com>
References: <CAD6PAKN_5WjWdQfonHLut9gSxD1b6DrToO5P+i5GW7=4q=V+bA@mail.gmail.com>
	<CAMYG4GntLCk9WCOE84tfMbPFG=xGc03DY5oZvWShvOm1grAuwQ@mail.gmail.com>
Message-ID: <526AB5EF.50107@gmail.com>

On 25/10/13 18:59, Matthew Knepley wrote:
> On Fri, Oct 25, 2013 at 11:33 AM, Torquil Macdonald S?rensen 
> <torquil at gmail.com <mailto:torquil at gmail.com>> wrote:
>
>     Hi!
>
>     I'm using PETSc 3.4.3, with only default runtime settings, to
>     solve Ax=b, for complex numbers (I have used
>     --with-scalar-type=complex).
>
>     After reading sections 4.3.1, 4.3.2, 4.3.3 in the manual, I
>     decided to investigate the convergence in my program, since I'm
>     having some problems when the system is larger. So, I'm running it
>     with the options
>
>     -ksp_monitor_true_residual -ksp_converged_reason -ksp_view
>
>     The output shows that the relative tolerance required for
>     convergence is 1e-5, but none of the reported norms are that
>     small. Despite this, convergence is reported (CONVERGED_RTOL).
>
>     I though that CONVERGED_RTOL should indicate that ||r||_2/||b||_2
>     < 1e-5, where r is the preconditioned residual?
>
>     Here is the output of my program (which includes some code to
>     compute the l_2 norm of b, just before KSPSolve is run):
>
>     *************************************************************
>     Norm of b: 0.179829
>       0 KSP preconditioned resid norm 2.508789303280e+04 true resid
>     norm 1.798290175843e-01 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 1.856991132478e+00 true resid
>     norm 3.186462510386e-01 ||r(i)||/||b|| 1.771940120227e+00
>       2 KSP preconditioned resid norm 1.704678606994e-01 true resid
>     norm 4.004071321891e-02 ||r(i)||/||b|| 2.226599119363e-01
>
>
> So the relative preconditioned residual is:
>
>   1.7e-01 / 2.5e+04 < 1.0e-5
>
> while the relative true residual is 2.2e-1. This happens because you 
> are using ILU(0). It can be really
> crappy, and in this case the preconditioner is very ill-conditioned. I 
> can't imagine a problem where I
> would recommend using ILU, but its our default because its the only 
> black-box PC that exists.
>
>   Thanks
>
>      Matt
>

 From your explanation I can see that I had misunderstood the meaning of 
"relative preconditioned residual norm", but now I understand. Thanks 
for clearing it up!

Best regards
Torquil S?rensen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/c6ffd650/attachment.html>

From bisheshkh at gmail.com  Fri Oct 25 14:55:50 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Fri, 25 Oct 2013 21:55:50 +0200
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
Message-ID: <CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>

On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> Dear all,
>> I would like to know if some of the petsc objects that I have not used so
>> far (IS, DMPlex, PetscSection) could be useful in the following case (of
>> irregular domains):
>>
>> Let's say that I have a 3D binary image (a cube).
>> The binary information of the image partitions the cube into a
>> computational domain and non-computational domain.
>> I must solve a pde (say a Poisson equation) only on the computational
>> domains (e.g: two isolated spheres within the cube). I'm using finite
>> difference and say a dirichlet boundary condition
>>
>> I know that I can create a dmda that will let me access the information
>> from this 3D binary image, get all the coefficients, rhs values etc using
>> the natural indexing (i,j,k).
>>
>> Now, I would like to create a matrix corresponding to the laplace
>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>> takes care of the dirchlet values too.
>> But in this matrix it should have the rows corresponding to the nodes
>> only on the computational domain. It would be nice if I can easily (using
>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>> boundary points.
>> Then, once the system is solved, put the values of the solution back to
>> the corresponding positions in the binary image.
>> Later, I might have to extend this for the staggered grid case too.
>> So is petscsection or dmplex suitable for this so that I can set up the
>> matrix with something like DMCreateMatrix ? Or what would you suggest as a
>> suitable approach to this problem ?
>>
>> I have looked at the manual and that led me to search for a simpler
>> examples in petsc src directories. But most of the ones I encountered are
>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>> more as a distraction with FEM jargon!)
>>
>
> It sounds like the right solution for this is to use PetscSection on top
> of DMDA. I am working on this, but it is really
> alpha code. If you feel comfortable with that level of development, we can
> help you.
>

Thanks, with the (short) experience of using Petsc so far and being
familiar with the awesomeness (quick and helpful replies) of this mailing
list, I would like to give it a try. Please give me some pointers to get
going for the example case I mentioned above. A simple example of using
PetscSection along with DMDA for finite volume (No FEM) would be great I
think.
Just a note: I'm currently using the petsc3.4.3 and have not used the
development version before.

If not, just put the identity into
> the rows you do not use on the full cube. It will not hurt scalability or
> convergence.
>

In the case of Poisson with Dirichlet condition this might be the case. But
is it always true that having identity rows in the system matrix will not
hurt convergence ? I thought otherwise for the following reasons:
1)  Having read Jed's answer here :
http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
2) Some observation I am getting (but I am still doing more experiments to
confirm) while solving my staggered-grid 3D stokes flow with schur
complement and using -pc_type gamg for A00 matrix. Putting the identity
rows for dirichlet boundaries and for ghost cells seemed to have effects on
its convergence. I'm hoping once I know how to use PetscSection, I can get
rid of using ghost cells method for the staggered grid and get rid of the
identity rows too.
Anyway please provide me with some pointers so that I can start trying with
petscsection on top of a dmda, in the beginning for non-staggered case.

Thanks,
Bishesh

>
>   Matt
>
>
>> Thanks,
>> Bishesh
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/b6794e12/attachment-0001.html>

From knepley at gmail.com  Fri Oct 25 15:21:57 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Oct 2013 15:21:57 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
Message-ID: <CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>

On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> Dear all,
>>> I would like to know if some of the petsc objects that I have not used
>>> so far (IS, DMPlex, PetscSection) could be useful in the following case (of
>>> irregular domains):
>>>
>>> Let's say that I have a 3D binary image (a cube).
>>> The binary information of the image partitions the cube into a
>>> computational domain and non-computational domain.
>>> I must solve a pde (say a Poisson equation) only on the computational
>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>> difference and say a dirichlet boundary condition
>>>
>>> I know that I can create a dmda that will let me access the information
>>> from this 3D binary image, get all the coefficients, rhs values etc using
>>> the natural indexing (i,j,k).
>>>
>>> Now, I would like to create a matrix corresponding to the laplace
>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>> takes care of the dirchlet values too.
>>> But in this matrix it should have the rows corresponding to the nodes
>>> only on the computational domain. It would be nice if I can easily (using
>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>> boundary points.
>>> Then, once the system is solved, put the values of the solution back to
>>> the corresponding positions in the binary image.
>>> Later, I might have to extend this for the staggered grid case too.
>>> So is petscsection or dmplex suitable for this so that I can set up the
>>> matrix with something like DMCreateMatrix ? Or what would you suggest as a
>>> suitable approach to this problem ?
>>>
>>> I have looked at the manual and that led me to search for a simpler
>>> examples in petsc src directories. But most of the ones I encountered are
>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>> more as a distraction with FEM jargon!)
>>>
>>
>> It sounds like the right solution for this is to use PetscSection on top
>> of DMDA. I am working on this, but it is really
>> alpha code. If you feel comfortable with that level of development, we
>> can help you.
>>
>
> Thanks, with the (short) experience of using Petsc so far and being
> familiar with the awesomeness (quick and helpful replies) of this mailing
> list, I would like to give it a try. Please give me some pointers to get
> going for the example case I mentioned above. A simple example of using
> PetscSection along with DMDA for finite volume (No FEM) would be great I
> think.
> Just a note: I'm currently using the petsc3.4.3 and have not used the
> development version before.
>

Okay,

1)  clone the repository using Git and build the 'next' branch.

2) then we will need to create a PetscSection that puts unknowns where you
want them

3) Setup the solver as usual

You can do 1) an 3) before we do 2).


 If not, just put the identity into
>> the rows you do not use on the full cube. It will not hurt scalability or
>> convergence.
>>
>
> In the case of Poisson with Dirichlet condition this might be the case.
> But is it always true that having identity rows in the system matrix will
> not hurt convergence ? I thought otherwise for the following reasons:
> 1)  Having read Jed's answer here :
> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>

Jed is talking about a constraint on a the pressure at a point. This is
just decoupling these unknowns from the rest
of the problem.


> 2) Some observation I am getting (but I am still doing more experiments to
> confirm) while solving my staggered-grid 3D stokes flow with schur
> complement and using -pc_type gamg for A00 matrix. Putting the identity
> rows for dirichlet boundaries and for ghost cells seemed to have effects on
> its convergence. I'm hoping once I know how to use PetscSection, I can get
> rid of using ghost cells method for the staggered grid and get rid of the
> identity rows too.
>

It can change the exact iteration, but it does not make the matrix
conditioning worse.

   Matt


> Anyway please provide me with some pointers so that I can start trying
> with petscsection on top of a dmda, in the beginning for non-staggered case.
>
> Thanks,
> Bishesh
>
>>
>>   Matt
>>
>>
>>> Thanks,
>>> Bishesh
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/b6ac01de/attachment.html>

From jedbrown at mcs.anl.gov  Fri Oct 25 15:04:54 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 25 Oct 2013 15:04:54 -0500
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
	<CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
Message-ID: <87txg53ymx.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> After playing around, I found something interesting, an option that I have
> never seen in any example: TSSetEquationType. I did a TSGetEquationType and
> I got TS_EQ_UNSPECIFIED. Then I thought, ok, let's try to specify the
> equation type and I added the following line to the code
>
>   ierr = TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1);

Thanks for tracking this down.  Indeed, that example enforces boundary
conditions using an algebraic expression:

    if (i == 0) {
      f[i].u = hx * (x[i].u - user->uleft);
      f[i].v = hx * (x[i].v - user->vleft);

For this reason, we can only obtain the stage derivative by
differencing, rather than by evaluating the function.

We need to audit the examples and use TSSetEquationType everywhere that
it's needed, which is probably in several of the examples.

Note that Dirichlet boundary conditions can also be chosen by setting an
appropriate initial condition and using the equation du/dt = 0.

> Well...the results is totally different, and in my opinion much more
> consistent. The value at the boundary is now the expected one:
>
> 0    10.000000     <-----
> 1    9.888647
> 2    9.777382
> 3    9.666293
> 4    9.555467
>
> ...
> ...
> 112    5.011530
> 113    5.010665
> 114    5.009880
> 115    5.009169
> ...
> ...
> 252    9.666293
> 253    9.777382
> 254    9.888647
> 255    10.000000     <----
>
>
> Moreover, I see much more effect of diffusion. Without this option,
> diffusion is always very weak, whatever the diffusion coefficient one puts,
> which always surprised me.
>
> What is the effect of the TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1); ?
>
> Christophe
>
>
> On Fri, Oct 25, 2013 at 11:47 AM, Christophe Ortiz <
> christophe.ortiz at ciemat.es> wrote:
>
>> Hi guys,
>>
>> I was having some troubles with my code (dof >> 1) and I decided to go
>> back to simple things. Therefore, I took ex25.c, which is a nice case for
>> me: diffusion of 2 species and 1 reaction.
>>
>> First, to have a better mesh I set 256 points (instead of the 11) in
>>
>>   ierr =
>> DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,256,2,2,NULL,&da);
>>
>> Then, I did something simple. I removed reaction terms by commenting
>>
>> ierr = TSSetRHSFunction(ts,NULL,FormRHSFunction,&user);CHKERRQ(ierr);
>>
>> So we are left with 2 diffusion equations:
>>
>> u_t -alpha u_xx = 0
>> v_t - alpha v_xx = 0
>>
>> Now, do something simple:
>> In FormInitialSolution, for variable v, change it and simply set it to 5
>> (for instance), ie, v will have uniform values from i=0 to i=mx-1.
>>
>> Now, for boundary conditions (Dirichlet here), set vleft and vright to 10,
>> for instance.
>>
>> Since values at the boundaries (10.0) are higher than values of initial
>> solution (5.0), what you expect is some diffusion towards the center from
>> the boundaries. Since we use Dirichlet boundary conditions here, you expect
>> the values of v at i=0 and i=mx-1 to be 10.0. It can't be something else.
>> Well, that's not the case. I see that the value of v at i=0 and at i=mx-1
>> is always the initial value (t=0). Here is a sample of the final values
>> after diffusion:
>>
>> 0    5.000000
>> 1    8.951153
>> 2    8.414078
>> 3    7.964193
>> 4    7.581196
>> ...
>> ...
>> 77    5.000235
>> 78    5.000207
>> 79    5.000182
>> 80    5.000161
>> ...
>> ...
>> 252    7.964193
>> 253    8.414078
>> 254    8.951153
>> 255    5.000000
>>
>> Then I checked the value of v inside IFunction at each timestep. I found
>> out that each time IFunction is called, the first value of v that is used
>> to evaluate F is the initial value (5.0), instead of of the value set by
>> Dirichlet (10). For next evaluations within the same timestep, it is 10.0,
>> the value imposed by Dirichlet boundary conditions.
>> Some output:
>>
>>       TSAdapt 'basic': step 324 accepted t=9.73302    + 9.496e-02
>> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.569e-02
>> 5.000000
>> 10.000000
>> 10.000000
>> 10.000000
>> 10.000000
>>
>>       TSAdapt 'basic': step 325 accepted t=9.82798    + 9.569e-02
>> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.642e-02
>> 5.000000
>> 10.000000
>> 10.000000
>> 10.000000
>> 10.000000
>>
>> Is it a bug or did I miss something ? I would expect that to evaluate F
>> within IFunction, it would always use the value set by Dirichlet.
>>
>>
>> I use petsc 3.4.1and configured it with
>>  --with-mpi=0 --with-fc=ifort --with-cc=icc --with-debugging=1
>> --with-scalar-type=real --with-precision=double
>> --with-blas-lapack-dir=/opt/intel/mkl.
>>
>> The options I used to run the example are
>> -ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
>> -ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
>> -snes_linesearch_type bt.
>>
>>
>> Christophe
>>
>>
>>
>>
>>
>>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/e46af289/attachment.pgp>

From mfadams at lbl.gov  Fri Oct 25 17:42:01 2013
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 25 Oct 2013 18:42:01 -0400
Subject: [petsc-users] PetscOptionsView() line 692
Message-ID: <CADOhEh55Zw-PXmRebKmHTsgPcc86GXKtLBeg-J9EecCcwGeyDQ@mail.gmail.com>

Has this been fixed in latter releases?  Would removing -options_left fix
this?

[0]PETSC ERROR: No support for this operation for this object type!
[0]PETSC ERROR: Only supports ASCII viewer!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.0, May, 13, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./driver2d.Linux.64.mpiCC.mpif90.DEBUG.MPI.PETSC.ex on a
arch-linux-gnu-anag-parallel-opt named loren by dmartin Fri Oct 25 15:20:25
2013
[0]PETSC ERROR: Libraries linked from
/usr/local/anag/pkg/petsc-3.4.0/arch-linux-gnu-anag-parallel-opt/lib
[0]PETSC ERROR: Configure run at Wed May 22 18:57:17 2013
[0]PETSC ERROR: Configure options --with-x=0 --download-hypre=1
--download-parmetis=1 --download-metis=1 --with-debugging=0 --with-mpi=1
PETSC_ARCH=arch-linux-gnu-anag-parallel-opt
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscOptionsView() line 692 in
/usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/options.c
[0]PETSC ERROR: PetscFinalize() line 1048 in
/usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/pinit.c
[0]PETSC ERROR: main() line 1141 in "unknowndirectory/"driver.cpp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/d97c57f1/attachment.html>

From balay at mcs.anl.gov  Fri Oct 25 19:31:35 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 25 Oct 2013 19:31:35 -0500 (CDT)
Subject: [petsc-users] PetscOptionsView() line 692
In-Reply-To: <CADOhEh55Zw-PXmRebKmHTsgPcc86GXKtLBeg-J9EecCcwGeyDQ@mail.gmail.com>
References: <CADOhEh55Zw-PXmRebKmHTsgPcc86GXKtLBeg-J9EecCcwGeyDQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310251929370.19655@mcs.anl.gov>

I can't reproduce this.

Is this reproduceable with the PETSc examples with your build?

Satish

On Fri, 25 Oct 2013, Mark Adams wrote:

> Has this been fixed in latter releases?  Would removing -options_left fix
> this?
> 
> [0]PETSC ERROR: No support for this operation for this object type!
> [0]PETSC ERROR: Only supports ASCII viewer!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.0, May, 13, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./driver2d.Linux.64.mpiCC.mpif90.DEBUG.MPI.PETSC.ex on a
> arch-linux-gnu-anag-parallel-opt named loren by dmartin Fri Oct 25 15:20:25
> 2013
> [0]PETSC ERROR: Libraries linked from
> /usr/local/anag/pkg/petsc-3.4.0/arch-linux-gnu-anag-parallel-opt/lib
> [0]PETSC ERROR: Configure run at Wed May 22 18:57:17 2013
> [0]PETSC ERROR: Configure options --with-x=0 --download-hypre=1
> --download-parmetis=1 --download-metis=1 --with-debugging=0 --with-mpi=1
> PETSC_ARCH=arch-linux-gnu-anag-parallel-opt
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscOptionsView() line 692 in
> /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/options.c
> [0]PETSC ERROR: PetscFinalize() line 1048 in
> /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/pinit.c
> [0]PETSC ERROR: main() line 1141 in "unknowndirectory/"driver.cpp
> 


From mfadams at lbl.gov  Fri Oct 25 19:41:52 2013
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 25 Oct 2013 20:41:52 -0400
Subject: [petsc-users] PetscOptionsView() line 692
In-Reply-To: <alpine.LFD.2.03.1310251929370.19655@mcs.anl.gov>
References: <CADOhEh55Zw-PXmRebKmHTsgPcc86GXKtLBeg-J9EecCcwGeyDQ@mail.gmail.com>
	<alpine.LFD.2.03.1310251929370.19655@mcs.anl.gov>
Message-ID: <CADOhEh77mkvpJ8LD5a-4DsWaBp10QXvsfDmEpeU5hovgCa+Krg@mail.gmail.com>

I have seen this several times over the past year or so.  It is in an
application.  It is at the end of the run and not using -options_left fixes
it.  I'm not set up on this machine and I don't have permissions to run a
test.  I have to think that there is some funny interaction with the way we
build or something.  So our user is happy now.  Lets just hope this problem
goes back to where it came from.


On Fri, Oct 25, 2013 at 8:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> I can't reproduce this.
>
> Is this reproduceable with the PETSc examples with your build?
>
> Satish
>
> On Fri, 25 Oct 2013, Mark Adams wrote:
>
> > Has this been fixed in latter releases?  Would removing -options_left fix
> > this?
> >
> > [0]PETSC ERROR: No support for this operation for this object type!
> > [0]PETSC ERROR: Only supports ASCII viewer!
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.0, May, 13, 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./driver2d.Linux.64.mpiCC.mpif90.DEBUG.MPI.PETSC.ex on a
> > arch-linux-gnu-anag-parallel-opt named loren by dmartin Fri Oct 25
> 15:20:25
> > 2013
> > [0]PETSC ERROR: Libraries linked from
> > /usr/local/anag/pkg/petsc-3.4.0/arch-linux-gnu-anag-parallel-opt/lib
> > [0]PETSC ERROR: Configure run at Wed May 22 18:57:17 2013
> > [0]PETSC ERROR: Configure options --with-x=0 --download-hypre=1
> > --download-parmetis=1 --download-metis=1 --with-debugging=0 --with-mpi=1
> > PETSC_ARCH=arch-linux-gnu-anag-parallel-opt
> > [0]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: PetscOptionsView() line 692 in
> > /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/options.c
> > [0]PETSC ERROR: PetscFinalize() line 1048 in
> > /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/pinit.c
> > [0]PETSC ERROR: main() line 1141 in "unknowndirectory/"driver.cpp
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/e2f3e0b0/attachment-0001.html>

From knepley at gmail.com  Fri Oct 25 19:46:46 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 Oct 2013 19:46:46 -0500
Subject: [petsc-users] PetscOptionsView() line 692
In-Reply-To: <CADOhEh77mkvpJ8LD5a-4DsWaBp10QXvsfDmEpeU5hovgCa+Krg@mail.gmail.com>
References: <CADOhEh55Zw-PXmRebKmHTsgPcc86GXKtLBeg-J9EecCcwGeyDQ@mail.gmail.com>
	<alpine.LFD.2.03.1310251929370.19655@mcs.anl.gov>
	<CADOhEh77mkvpJ8LD5a-4DsWaBp10QXvsfDmEpeU5hovgCa+Krg@mail.gmail.com>
Message-ID: <CAMYG4Gn3AT7k9aM0NY5mgtgb=H8A-Uy0wN8JmUP7FZbHV0S+uQ@mail.gmail.com>

On Fri, Oct 25, 2013 at 7:41 PM, Mark Adams <mfadams at lbl.gov> wrote:

> I have seen this several times over the past year or so.  It is in an
> application.  It is at the end of the run and not using -options_left fixes
> it.  I'm not set up on this machine and I don't have permissions to run a
> test.  I have to think that there is some funny interaction with the way we
> build or something.  So our user is happy now.  Lets just hope this problem
> goes back to where it came from.
>

This is real. I had it. Looking...

   Matt


> On Fri, Oct 25, 2013 at 8:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> I can't reproduce this.
>>
>> Is this reproduceable with the PETSc examples with your build?
>>
>> Satish
>>
>> On Fri, 25 Oct 2013, Mark Adams wrote:
>>
>> > Has this been fixed in latter releases?  Would removing -options_left
>> fix
>> > this?
>> >
>> > [0]PETSC ERROR: No support for this operation for this object type!
>> > [0]PETSC ERROR: Only supports ASCII viewer!
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: Petsc Release Version 3.4.0, May, 13, 2013
>> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> > [0]PETSC ERROR: See docs/index.html for manual pages.
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: ./driver2d.Linux.64.mpiCC.mpif90.DEBUG.MPI.PETSC.ex on a
>> > arch-linux-gnu-anag-parallel-opt named loren by dmartin Fri Oct 25
>> 15:20:25
>> > 2013
>> > [0]PETSC ERROR: Libraries linked from
>> > /usr/local/anag/pkg/petsc-3.4.0/arch-linux-gnu-anag-parallel-opt/lib
>> > [0]PETSC ERROR: Configure run at Wed May 22 18:57:17 2013
>> > [0]PETSC ERROR: Configure options --with-x=0 --download-hypre=1
>> > --download-parmetis=1 --download-metis=1 --with-debugging=0 --with-mpi=1
>> > PETSC_ARCH=arch-linux-gnu-anag-parallel-opt
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: PetscOptionsView() line 692 in
>> > /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/options.c
>> > [0]PETSC ERROR: PetscFinalize() line 1048 in
>> > /usr/local/anag/pkg/petsc-3.4.0/src/sys/objects/pinit.c
>> > [0]PETSC ERROR: main() line 1141 in "unknowndirectory/"driver.cpp
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131025/efa5f86d/attachment.html>

From bisheshkh at gmail.com  Sat Oct 26 03:12:03 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Sat, 26 Oct 2013 10:12:03 +0200
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
Message-ID: <CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>

On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> Dear all,
>>>> I would like to know if some of the petsc objects that I have not used
>>>> so far (IS, DMPlex, PetscSection) could be useful in the following case (of
>>>> irregular domains):
>>>>
>>>> Let's say that I have a 3D binary image (a cube).
>>>> The binary information of the image partitions the cube into a
>>>> computational domain and non-computational domain.
>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>> difference and say a dirichlet boundary condition
>>>>
>>>> I know that I can create a dmda that will let me access the information
>>>> from this 3D binary image, get all the coefficients, rhs values etc using
>>>> the natural indexing (i,j,k).
>>>>
>>>> Now, I would like to create a matrix corresponding to the laplace
>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>> takes care of the dirchlet values too.
>>>> But in this matrix it should have the rows corresponding to the nodes
>>>> only on the computational domain. It would be nice if I can easily (using
>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>> boundary points.
>>>> Then, once the system is solved, put the values of the solution back to
>>>> the corresponding positions in the binary image.
>>>> Later, I might have to extend this for the staggered grid case too.
>>>> So is petscsection or dmplex suitable for this so that I can set up the
>>>> matrix with something like DMCreateMatrix ? Or what would you suggest as a
>>>> suitable approach to this problem ?
>>>>
>>>> I have looked at the manual and that led me to search for a simpler
>>>> examples in petsc src directories. But most of the ones I encountered are
>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>> more as a distraction with FEM jargon!)
>>>>
>>>
>>> It sounds like the right solution for this is to use PetscSection on top
>>> of DMDA. I am working on this, but it is really
>>> alpha code. If you feel comfortable with that level of development, we
>>> can help you.
>>>
>>
>> Thanks, with the (short) experience of using Petsc so far and being
>> familiar with the awesomeness (quick and helpful replies) of this mailing
>> list, I would like to give it a try. Please give me some pointers to get
>> going for the example case I mentioned above. A simple example of using
>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>> think.
>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>> development version before.
>>
>
> Okay,
>
> 1)  clone the repository using Git and build the 'next' branch.
>

I encountered errors when doing make on the 'next' branch. The errors are
as follows (I tried attached the configure.log file but the email bounced
back saying it awaits moderator approval for having too big an attachment,
so I'm sending this one with only make.log attached. ) :

         CXX arch-linux2-cxx-debug/obj/src/
mat/order/ftn-auto/spectralf.o
         CXX arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
src/mat/order/wbm.c: In function ?PetscErrorCode MatGetOrdering_WBM(Mat,
MatOrderingType, _p_IS**, _p_IS**)?:
src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
[-Wunused-but-set-variable]
src/mat/order/wbm.c:15:36: warning: unused variable ?num?
[-Wunused-variable]
src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
[-Wunused-but-set-variable]
src/mat/order/wbm.c:15:66: warning: unused variable ?info?
[-Wunused-variable]
         CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
src/mat/order/ftn-auto/spectralf.c: In function ?void
matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian? was
not declared in this scope
gmake[2]: ***
[arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not used
[-Wunused-but-set-variable]
src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not used
[-Wunused-but-set-variable]
src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not used
[-Wunused-but-set-variable]
gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
gmake[1]: *** [gnumake] Error 2
gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
**************************ERROR*************************************
  Error during compile, check arch-linux2-cxx-debug/conf/make.log
  Send it and arch-linux2-cxx-debug/conf/configure.log to
petsc-maint at mcs.anl.gov
********************************************************************
make: *** [all] Error 1



>
> 2) then we will need to create a PetscSection that puts unknowns where you
> want them
>
> 3) Setup the solver as usual
>
> You can do 1) an 3) before we do 2).
>
>
>  If not, just put the identity into
>>> the rows you do not use on the full cube. It will not hurt scalability
>>> or convergence.
>>>
>>
>> In the case of Poisson with Dirichlet condition this might be the case.
>> But is it always true that having identity rows in the system matrix will
>> not hurt convergence ? I thought otherwise for the following reasons:
>> 1)  Having read Jed's answer here :
>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>
>
> Jed is talking about a constraint on a the pressure at a point. This is
> just decoupling these unknowns from the rest
> of the problem.
>
>
>> 2) Some observation I am getting (but I am still doing more experiments
>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>> rid of using ghost cells method for the staggered grid and get rid of the
>> identity rows too.
>>
>
> It can change the exact iteration, but it does not make the matrix
> conditioning worse.
>
>    Matt
>
>
>>  Anyway please provide me with some pointers so that I can start trying
>> with petscsection on top of a dmda, in the beginning for non-staggered case.
>>
>> Thanks,
>> Bishesh
>>
>>>
>>>   Matt
>>>
>>>
>>>> Thanks,
>>>> Bishesh
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131026/1d0039b6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: text/x-log
Size: 58653 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131026/1d0039b6/attachment-0001.bin>

From knepley at gmail.com  Sat Oct 26 07:57:44 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 26 Oct 2013 07:57:44 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
Message-ID: <CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>

On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> Dear all,
>>>>> I would like to know if some of the petsc objects that I have not used
>>>>> so far (IS, DMPlex, PetscSection) could be useful in the following case (of
>>>>> irregular domains):
>>>>>
>>>>> Let's say that I have a 3D binary image (a cube).
>>>>> The binary information of the image partitions the cube into a
>>>>> computational domain and non-computational domain.
>>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>>> difference and say a dirichlet boundary condition
>>>>>
>>>>> I know that I can create a dmda that will let me access the
>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>> etc using the natural indexing (i,j,k).
>>>>>
>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>> takes care of the dirchlet values too.
>>>>> But in this matrix it should have the rows corresponding to the nodes
>>>>> only on the computational domain. It would be nice if I can easily (using
>>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>>> boundary points.
>>>>> Then, once the system is solved, put the values of the solution back
>>>>> to the corresponding positions in the binary image.
>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>> as a suitable approach to this problem ?
>>>>>
>>>>> I have looked at the manual and that led me to search for a simpler
>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>> more as a distraction with FEM jargon!)
>>>>>
>>>>
>>>> It sounds like the right solution for this is to use PetscSection on
>>>> top of DMDA. I am working on this, but it is really
>>>> alpha code. If you feel comfortable with that level of development, we
>>>> can help you.
>>>>
>>>
>>> Thanks, with the (short) experience of using Petsc so far and being
>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>> list, I would like to give it a try. Please give me some pointers to get
>>> going for the example case I mentioned above. A simple example of using
>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>> think.
>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>> development version before.
>>>
>>
>> Okay,
>>
>> 1)  clone the repository using Git and build the 'next' branch.
>>
>
> I encountered errors when doing make on the 'next' branch. The errors are
> as follows (I tried attached the configure.log file but the email bounced
> back saying it awaits moderator approval for having too big an attachment,
> so I'm sending this one with only make.log attached. ) :
>

They are fixed. Pull again and rebuild.

  Matt


>          CXX arch-linux2-cxx-debug/obj/src/
> mat/order/ftn-auto/spectralf.o
>          CXX arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
> src/mat/order/wbm.c: In function ?PetscErrorCode MatGetOrdering_WBM(Mat,
> MatOrderingType, _p_IS**, _p_IS**)?:
> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
> [-Wunused-but-set-variable]
> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
> [-Wunused-variable]
> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
> [-Wunused-but-set-variable]
> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
> [-Wunused-variable]
>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
> src/mat/order/ftn-auto/spectralf.c: In function ?void
> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian? was
> not declared in this scope
> gmake[2]: ***
> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
> gmake[2]: *** Waiting for unfinished jobs....
> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not used
> [-Wunused-but-set-variable]
> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not used
> [-Wunused-but-set-variable]
> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not used
> [-Wunused-but-set-variable]
> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
> gmake[1]: *** [gnumake] Error 2
> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
> **************************ERROR*************************************
>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>   Send it and arch-linux2-cxx-debug/conf/configure.log to
> petsc-maint at mcs.anl.gov
> ********************************************************************
> make: *** [all] Error 1
>
>
>
>>
>> 2) then we will need to create a PetscSection that puts unknowns where
>> you want them
>>
>> 3) Setup the solver as usual
>>
>> You can do 1) an 3) before we do 2).
>>
>>
>>  If not, just put the identity into
>>>> the rows you do not use on the full cube. It will not hurt scalability
>>>> or convergence.
>>>>
>>>
>>> In the case of Poisson with Dirichlet condition this might be the case.
>>> But is it always true that having identity rows in the system matrix will
>>> not hurt convergence ? I thought otherwise for the following reasons:
>>> 1)  Having read Jed's answer here :
>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>
>>
>> Jed is talking about a constraint on a the pressure at a point. This is
>> just decoupling these unknowns from the rest
>> of the problem.
>>
>>
>>> 2) Some observation I am getting (but I am still doing more experiments
>>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>>> rid of using ghost cells method for the staggered grid and get rid of the
>>> identity rows too.
>>>
>>
>> It can change the exact iteration, but it does not make the matrix
>> conditioning worse.
>>
>>    Matt
>>
>>
>>>  Anyway please provide me with some pointers so that I can start trying
>>> with petscsection on top of a dmda, in the beginning for non-staggered case.
>>>
>>> Thanks,
>>> Bishesh
>>>
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Bishesh
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131026/b5c515ce/attachment.html>

From bisheshkh at gmail.com  Sat Oct 26 08:27:25 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Sat, 26 Oct 2013 15:27:25 +0200
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
	<CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
Message-ID: <CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>

On Sat, Oct 26, 2013 at 2:57 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> Dear all,
>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>> case (of irregular domains):
>>>>>>
>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>> The binary information of the image partitions the cube into a
>>>>>> computational domain and non-computational domain.
>>>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>>>> difference and say a dirichlet boundary condition
>>>>>>
>>>>>> I know that I can create a dmda that will let me access the
>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>> etc using the natural indexing (i,j,k).
>>>>>>
>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>> takes care of the dirchlet values too.
>>>>>> But in this matrix it should have the rows corresponding to the nodes
>>>>>> only on the computational domain. It would be nice if I can easily (using
>>>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>>>> boundary points.
>>>>>> Then, once the system is solved, put the values of the solution back
>>>>>> to the corresponding positions in the binary image.
>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>> as a suitable approach to this problem ?
>>>>>>
>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>> more as a distraction with FEM jargon!)
>>>>>>
>>>>>
>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>> top of DMDA. I am working on this, but it is really
>>>>> alpha code. If you feel comfortable with that level of development, we
>>>>> can help you.
>>>>>
>>>>
>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>> list, I would like to give it a try. Please give me some pointers to get
>>>> going for the example case I mentioned above. A simple example of using
>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>> think.
>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>> development version before.
>>>>
>>>
>>> Okay,
>>>
>>> 1)  clone the repository using Git and build the 'next' branch.
>>>
>>
>> I encountered errors when doing make on the 'next' branch. The errors are
>> as follows (I tried attached the configure.log file but the email bounced
>> back saying it awaits moderator approval for having too big an attachment,
>> so I'm sending this one with only make.log attached. ) :
>>
>
> They are fixed. Pull again and rebuild.
>

doing git pull in the next branch says "Already up-to-date.", I'm not sure
if it should have said that then. Still tried ./configure and make again
but returns the same error. Does it take sometime to get updated in the
servers or do I need to do anything special again other than the following
? :
git checkout next
git pull



>   Matt
>
>
>>           CXX arch-linux2-cxx-debug/obj/src/
>> mat/order/ftn-auto/spectralf.o
>>          CXX arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
>> src/mat/order/wbm.c: In function ?PetscErrorCode MatGetOrdering_WBM(Mat,
>> MatOrderingType, _p_IS**, _p_IS**)?:
>> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
>> [-Wunused-but-set-variable]
>> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
>> [-Wunused-variable]
>> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
>> [-Wunused-but-set-variable]
>> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
>> [-Wunused-variable]
>>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
>> src/mat/order/ftn-auto/spectralf.c: In function ?void
>> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
>> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian? was
>> not declared in this scope
>> gmake[2]: ***
>> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
>> gmake[2]: *** Waiting for unfinished jobs....
>> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
>> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
>> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
>> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
>> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not
>> used [-Wunused-but-set-variable]
>> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not
>> used [-Wunused-but-set-variable]
>> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not
>> used [-Wunused-but-set-variable]
>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>> gmake[1]: *** [gnumake] Error 2
>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>> **************************ERROR*************************************
>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>> petsc-maint at mcs.anl.gov
>> ********************************************************************
>> make: *** [all] Error 1
>>
>>
>>
>>>
>>> 2) then we will need to create a PetscSection that puts unknowns where
>>> you want them
>>>
>>> 3) Setup the solver as usual
>>>
>>> You can do 1) an 3) before we do 2).
>>>
>>>
>>>  If not, just put the identity into
>>>>> the rows you do not use on the full cube. It will not hurt scalability
>>>>> or convergence.
>>>>>
>>>>
>>>> In the case of Poisson with Dirichlet condition this might be the case.
>>>> But is it always true that having identity rows in the system matrix will
>>>> not hurt convergence ? I thought otherwise for the following reasons:
>>>> 1)  Having read Jed's answer here :
>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>
>>>
>>> Jed is talking about a constraint on a the pressure at a point. This is
>>> just decoupling these unknowns from the rest
>>> of the problem.
>>>
>>>
>>>> 2) Some observation I am getting (but I am still doing more experiments
>>>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>>>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>>>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>>>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>>>> rid of using ghost cells method for the staggered grid and get rid of the
>>>> identity rows too.
>>>>
>>>
>>> It can change the exact iteration, but it does not make the matrix
>>> conditioning worse.
>>>
>>>    Matt
>>>
>>>
>>>>  Anyway please provide me with some pointers so that I can start
>>>> trying with petscsection on top of a dmda, in the beginning for
>>>> non-staggered case.
>>>>
>>>> Thanks,
>>>> Bishesh
>>>>
>>>>>
>>>>>   Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Bishesh
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131026/7a97f207/attachment-0001.html>

From torquil at gmail.com  Sat Oct 26 15:05:04 2013
From: torquil at gmail.com (=?UTF-8?Q?Torquil_Macdonald_S=C3=B8rensen?=)
Date: Sat, 26 Oct 2013 22:05:04 +0200
Subject: [petsc-users] ILUTP
Message-ID: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>

Hi!

I have a linear problem Ax=b, with complex values, that is solved very well
using the ILUTP and GMRES implementations in GMM++. None of the other
preconditioners available in GMM++ would work (e.g. ILU and ILUT) for this
problem.

Then I tried the same problem using ILU and GMRES in PETSc, with no
success, despite a lot of attempts at adjusting ILU settings. I always end
up with gigantic residual norm values. The same PETSc program works well
when I apply it to a different matrix A.

I'm now suspecting that the ILU options cannot be set so as to obtain ILUTP
behaviour.

What would be the recommended method to access an ILUTP preconditioner from
PETSc?

According to the PETSc website, a preconditioner named ILUDT is available
by using the external package Hypre, but I had to deselect Hypre during the
PETSc build due to my use of complex numbers... So do you guys think that I
should transform everything to a real representation and try Hypre/ILUDT?

Best regards
Torquil S?rensen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131026/b1bdd4fc/attachment.html>

From talebi.hossein at gmail.com  Sun Oct 27 06:16:04 2013
From: talebi.hossein at gmail.com (Hossein Talebi)
Date: Sun, 27 Oct 2013 12:16:04 +0100
Subject: [petsc-users] Fortran interface -- low level access to petsc data
Message-ID: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>

 Hello All,

I am trying to use petsc with Fortran. The initial mechanism to use petsc
with FORTRAN77 that works fine. However, I found the Fortran 90 interfaces
difficult to work with. The very first issue is one has to recompile petsc
with the Fortran compiler in case different to the one which was used to
create the Fortran Mod files. Also, we would still need to use C type
#include statements along with Fortran 'use' statement which is not so
optimal.

Since Fortran 2003 offers great interface to C, did anyone try to write a
F03 interface to petsc? Using F03 features, we can have very low data
access to petsc database as well.


Cheers
Hossein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/cd32c4be/attachment.html>

From knepley at gmail.com  Sun Oct 27 07:56:29 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 27 Oct 2013 07:56:29 -0500
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
Message-ID: <CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>

On Sun, Oct 27, 2013 at 6:16 AM, Hossein Talebi <talebi.hossein at gmail.com>wrote:

>  Hello All,
>
> I am trying to use petsc with Fortran. The initial mechanism to use petsc
> with FORTRAN77 that works fine. However, I found the Fortran 90 interfaces
> difficult to work with. The very first issue is one has to recompile petsc
> with the Fortran compiler in case different to the one which was used to
> create the Fortran Mod files. Also, we would still need to use C type
> #include statements along with Fortran 'use' statement which is not so
> optimal.
>
> Since Fortran 2003 offers great interface to C, did anyone try to write a
> F03 interface to petsc? Using F03 features, we can have very low data
> access to petsc database as well.
>

F2003 support is fairly thin in the compiler world, and we have not had
much call for it. We do gratefully accept contributions.

  Thanks,

     Matt


> Cheers
> Hossein
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/2295becc/attachment.html>

From knepley at gmail.com  Sun Oct 27 08:48:56 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 27 Oct 2013 08:48:56 -0500
Subject: [petsc-users] ILUTP
In-Reply-To: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
Message-ID: <CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>

On Sat, Oct 26, 2013 at 3:05 PM, Torquil Macdonald S?rensen <
torquil at gmail.com> wrote:

> Hi!
>
> I have a linear problem Ax=b, with complex values, that is solved very
> well using the ILUTP and GMRES implementations in GMM++. None of the other
> preconditioners available in GMM++ would work (e.g. ILU and ILUT) for this
> problem.
>
> Then I tried the same problem using ILU and GMRES in PETSc, with no
> success, despite a lot of attempts at adjusting ILU settings. I always end
> up with gigantic residual norm values. The same PETSc program works well
> when I apply it to a different matrix A.
>
> I'm now suspecting that the ILU options cannot be set so as to obtain
> ILUTP behaviour.
>
> What would be the recommended method to access an ILUTP preconditioner
> from PETSc?
>
> According to the PETSc website, a preconditioner named ILUDT is available
> by using the external package Hypre, but I had to deselect Hypre during the
> PETSc build due to my use of complex numbers... So do you guys think that I
> should transform everything to a real representation and try Hypre/ILUDT?
>

I suggest you look for a different preconditioner. The fact that you have
so much trouble reproducing the behavior shows
you just how fragile the performance of ILU is. It may work for a certain
size, but fail for larger or smaller problems, or slightly
different parameters. What problem are you solving? Usually the best option
is to consult the literature for preconditioners
tailored to your problem, and then reproduce them.

   Matt


> Best regards
> Torquil S?rensen
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/66d73481/attachment.html>

From knepley at gmail.com  Sun Oct 27 08:53:22 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 27 Oct 2013 08:53:22 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
	<CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
	<CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>
Message-ID: <CAMYG4Gkh5a5Htu-LM=+2ztu7M58PK0bPGdCBeFY4jdFkW=Lq6Q@mail.gmail.com>

On Sat, Oct 26, 2013 at 8:27 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Sat, Oct 26, 2013 at 2:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>>
>>>
>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>> case (of irregular domains):
>>>>>>>
>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>> The binary information of the image partitions the cube into a
>>>>>>> computational domain and non-computational domain.
>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>
>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>
>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>> takes care of the dirchlet values too.
>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>> the boundary points.
>>>>>>> Then, once the system is solved, put the values of the solution back
>>>>>>> to the corresponding positions in the binary image.
>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>>> as a suitable approach to this problem ?
>>>>>>>
>>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>>> more as a distraction with FEM jargon!)
>>>>>>>
>>>>>>
>>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>>> top of DMDA. I am working on this, but it is really
>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>> we can help you.
>>>>>>
>>>>>
>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>> going for the example case I mentioned above. A simple example of using
>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>> think.
>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>>> development version before.
>>>>>
>>>>
>>>> Okay,
>>>>
>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>
>>>
>>> I encountered errors when doing make on the 'next' branch. The errors
>>> are as follows (I tried attached the configure.log file but the email
>>> bounced back saying it awaits moderator approval for having too big an
>>> attachment, so I'm sending this one with only make.log attached. ) :
>>>
>>
>> They are fixed. Pull again and rebuild.
>>
>
> doing git pull in the next branch says "Already up-to-date.", I'm not sure
> if it should have said that then. Still tried ./configure and make again
> but returns the same error. Does it take sometime to get updated in the
> servers or do I need to do anything special again other than the following
> ? :
> git checkout next
> git pull
>

Okay, it should be this

git checkout next
git pull
make allfortranstubs
make

  Thanks,

     Matt


>
>>   Matt
>>
>>
>>>           CXX arch-linux2-cxx-debug/obj/src/
>>> mat/order/ftn-auto/spectralf.o
>>>          CXX
>>> arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
>>> src/mat/order/wbm.c: In function ?PetscErrorCode MatGetOrdering_WBM(Mat,
>>> MatOrderingType, _p_IS**, _p_IS**)?:
>>> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
>>> [-Wunused-but-set-variable]
>>> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
>>> [-Wunused-variable]
>>> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
>>> [-Wunused-but-set-variable]
>>> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
>>> [-Wunused-variable]
>>>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
>>> src/mat/order/ftn-auto/spectralf.c: In function ?void
>>> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
>>> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian?
>>> was not declared in this scope
>>> gmake[2]: ***
>>> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
>>> gmake[2]: *** Waiting for unfinished jobs....
>>> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
>>> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
>>> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
>>> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
>>> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not
>>> used [-Wunused-but-set-variable]
>>> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not
>>> used [-Wunused-but-set-variable]
>>> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not
>>> used [-Wunused-but-set-variable]
>>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>> gmake[1]: *** [gnumake] Error 2
>>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>> **************************ERROR*************************************
>>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>>> petsc-maint at mcs.anl.gov
>>> ********************************************************************
>>> make: *** [all] Error 1
>>>
>>>
>>>
>>>>
>>>> 2) then we will need to create a PetscSection that puts unknowns where
>>>> you want them
>>>>
>>>> 3) Setup the solver as usual
>>>>
>>>> You can do 1) an 3) before we do 2).
>>>>
>>>>
>>>>  If not, just put the identity into
>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>> scalability or convergence.
>>>>>>
>>>>>
>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>> case. But is it always true that having identity rows in the system matrix
>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>> 1)  Having read Jed's answer here :
>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>
>>>>
>>>> Jed is talking about a constraint on a the pressure at a point. This is
>>>> just decoupling these unknowns from the rest
>>>> of the problem.
>>>>
>>>>
>>>>> 2) Some observation I am getting (but I am still doing more
>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>> rid of the identity rows too.
>>>>>
>>>>
>>>> It can change the exact iteration, but it does not make the matrix
>>>> conditioning worse.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>>  Anyway please provide me with some pointers so that I can start
>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>> non-staggered case.
>>>>>
>>>>> Thanks,
>>>>> Bishesh
>>>>>
>>>>>>
>>>>>>   Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Bishesh
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/1a70457e/attachment-0001.html>

From bisheshkh at gmail.com  Sun Oct 27 10:04:44 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Sun, 27 Oct 2013 16:04:44 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4Gkh5a5Htu-LM=+2ztu7M58PK0bPGdCBeFY4jdFkW=Lq6Q@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
	<CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
	<CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>
	<CAMYG4Gkh5a5Htu-LM=+2ztu7M58PK0bPGdCBeFY4jdFkW=Lq6Q@mail.gmail.com>
Message-ID: <CAEhex8iVYObVjj2h3H5P3PO7ndFcz-mWxTLUY8qkrTRWNnHW9A@mail.gmail.com>

On Sun, Oct 27, 2013 at 2:53 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Oct 26, 2013 at 8:27 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Sat, Oct 26, 2013 at 2:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>> case (of irregular domains):
>>>>>>>>
>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>> computational domain and non-computational domain.
>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>
>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>
>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>> takes care of the dirchlet values too.
>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>> the boundary points.
>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>>>> as a suitable approach to this problem ?
>>>>>>>>
>>>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>>>> more as a distraction with FEM jargon!)
>>>>>>>>
>>>>>>>
>>>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>>>> top of DMDA. I am working on this, but it is really
>>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>>> we can help you.
>>>>>>>
>>>>>>
>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>> think.
>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>>>> development version before.
>>>>>>
>>>>>
>>>>> Okay,
>>>>>
>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>
>>>>
>>>> I encountered errors when doing make on the 'next' branch. The errors
>>>> are as follows (I tried attached the configure.log file but the email
>>>> bounced back saying it awaits moderator approval for having too big an
>>>> attachment, so I'm sending this one with only make.log attached. ) :
>>>>
>>>
>>> They are fixed. Pull again and rebuild.
>>>
>>
>> doing git pull in the next branch says "Already up-to-date.", I'm not
>> sure if it should have said that then. Still tried ./configure and make
>> again but returns the same error. Does it take sometime to get updated in
>> the servers or do I need to do anything special again other than the
>> following ? :
>> git checkout next
>> git pull
>>
>
> Okay, it should be this
>
> git checkout next
> git pull
> make allfortranstubs
> make
>

Thanks, but now it reports another error:

         CXX arch-linux2-cxx-debug/obj/src/dm/impls/redundant/dmredundant.o
         CXX
arch-linux2-cxx-debug/obj/src/dm/impls/redundant/ftn-auto/dmredundantf.o
         CXX arch-linux2-cxx-debug/obj/src/dm/impls/plex/plexcreate.o
src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
MatSetupDM_HYPREStruct(Mat, DM)?:
src/dm/impls/da/hypre/mhyp.c:485:58: error: invalid conversion from
?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
[-fpermissive]
/home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27: error:
initializing argument 3 of ?PetscErrorCode DMDAGetGlobalIndices(DM,
PetscInt*, const PetscInt**)? [-fpermissive]
src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
MatSetupDM_HYPRESStruct(Mat, DM)?:
src/dm/impls/da/hypre/mhyp.c:963:58: error: invalid conversion from
?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
[-fpermissive]
/home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27: error:
initializing argument 3 of ?PetscErrorCode DMDAGetGlobalIndices(DM,
PetscInt*, const PetscInt**)? [-fpermissive]
gmake[2]: *** [arch-linux2-cxx-debug/obj/src/dm/impls/da/hypre/mhyp.o]
Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
gmake[1]: *** [gnumake] Error 2
gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
**************************ERROR*************************************
  Error during compile, check arch-linux2-cxx-debug/conf/make.log
  Send it and arch-linux2-cxx-debug/conf/configure.log to
petsc-maint at mcs.anl.gov
********************************************************************
make: *** [all] Error 1


>
>   Thanks,
>
>      Matt
>
>
>>
>>>   Matt
>>>
>>>
>>>>           CXX arch-linux2-cxx-debug/obj/src/
>>>> mat/order/ftn-auto/spectralf.o
>>>>          CXX
>>>> arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
>>>> src/mat/order/wbm.c: In function ?PetscErrorCode
>>>> MatGetOrdering_WBM(Mat, MatOrderingType, _p_IS**, _p_IS**)?:
>>>> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
>>>> [-Wunused-but-set-variable]
>>>> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
>>>> [-Wunused-variable]
>>>> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
>>>> [-Wunused-but-set-variable]
>>>> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
>>>> [-Wunused-variable]
>>>>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
>>>> src/mat/order/ftn-auto/spectralf.c: In function ?void
>>>> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
>>>> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian?
>>>> was not declared in this scope
>>>> gmake[2]: ***
>>>> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
>>>> gmake[2]: *** Waiting for unfinished jobs....
>>>> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
>>>> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
>>>> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
>>>> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
>>>> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not
>>>> used [-Wunused-but-set-variable]
>>>> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not
>>>> used [-Wunused-but-set-variable]
>>>> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not
>>>> used [-Wunused-but-set-variable]
>>>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>> gmake[1]: *** [gnumake] Error 2
>>>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>> **************************ERROR*************************************
>>>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>>>> petsc-maint at mcs.anl.gov
>>>> ********************************************************************
>>>> make: *** [all] Error 1
>>>>
>>>>
>>>>
>>>>>
>>>>> 2) then we will need to create a PetscSection that puts unknowns where
>>>>> you want them
>>>>>
>>>>> 3) Setup the solver as usual
>>>>>
>>>>> You can do 1) an 3) before we do 2).
>>>>>
>>>>>
>>>>>  If not, just put the identity into
>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>> scalability or convergence.
>>>>>>>
>>>>>>
>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>> 1)  Having read Jed's answer here :
>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>
>>>>>
>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>> is just decoupling these unknowns from the rest
>>>>> of the problem.
>>>>>
>>>>>
>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>> rid of the identity rows too.
>>>>>>
>>>>>
>>>>> It can change the exact iteration, but it does not make the matrix
>>>>> conditioning worse.
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>> non-staggered case.
>>>>>>
>>>>>> Thanks,
>>>>>> Bishesh
>>>>>>
>>>>>>>
>>>>>>>   Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bishesh
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/8d48ab47/attachment.html>

From balay at mcs.anl.gov  Sun Oct 27 10:42:07 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Sun, 27 Oct 2013 10:42:07 -0500 (CDT)
Subject: [petsc-users] Fortran interface -- low level access to petsc
 data
In-Reply-To: <CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
Message-ID: <alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>

On Sun, 27 Oct 2013, Matthew Knepley wrote:

> On Sun, Oct 27, 2013 at 6:16 AM, Hossein Talebi <talebi.hossein at gmail.com>wrote:
> 
> >  Hello All,
> >
> > I am trying to use petsc with Fortran. The initial mechanism to use petsc
> > with FORTRAN77 that works fine. However, I found the Fortran 90 interfaces
> > difficult to work with. The very first issue is one has to recompile petsc
> > with the Fortran compiler in case different to the one which was used to
> > create the Fortran Mod files.

Yes - one cannot mix fortran code compiled with 2 different compilers
- or 2 different version of the same compiler wrt modules.

But we generally recommend not mixing compilers   for C aswell as       this
can cause problems.

Also you don't need to use modules to get f90 interfaces. You can include
the f90 interface include files    directly in your code.

#incude "finclude/petsc.h90"

> > Also, we would still need to use C type
> > #include statements along with Fortran 'use' statement which is not so
> > optimal.

Even if you were to use f90 modules #include is unavoidable as we still
have other stuff in include files.

check include/finclude/petscsysdef.h  include/finclude/petscsys.h

> > Since Fortran 2003 offers great interface to C, did anyone try to write a
> > F03 interface to petsc? Using F03 features, we can have very low data
> > access to petsc database as well.
> >
> 
> F2003 support is fairly thin in the compiler world, and we have not had
> much call for it. We do gratefully accept contributions.

Currently we have the f90 interfaces [in include/finclude/ftn-auto/]
automatically generated by bfort tool from the C interface. So to
support F03 it would be ideal if there is such a tool. [or update
bfort to do it]

Satish

From talebi.hossein at gmail.com  Sun Oct 27 12:36:18 2013
From: talebi.hossein at gmail.com (Hossein Talebi)
Date: Sun, 27 Oct 2013 18:36:18 +0100
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
Message-ID: <CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>

Thank you for your answer.

It is not true that F2003 support is that limited: Gfortran, Intel Fortran,
Cray Fortran, IBM Fortran and PGI Fortran all support nearly full F2003.
Some other ones support the C portability of F2003 although not supporting
the rest.

* I can contribute on creating this interface. I have experience with F2003
to C++ and it works fine while C is a lot easier. My idea is to create the
F03 interface to petsc as a separate tool. So, anyone trying to use petsc
with a F03 compiler can compile this interface again to avoid the compiler
inconsistency of Modules.

We can work on a python (or similar) script to generate the interfaces. For
example, a project called gtk-fortran uses this method. Can any of you guys
help a little for planning the interface design?


Cheers
Hossein












On Sun, Oct 27, 2013 at 4:42 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Sun, 27 Oct 2013, Matthew Knepley wrote:
>
> > On Sun, Oct 27, 2013 at 6:16 AM, Hossein Talebi <
> talebi.hossein at gmail.com>wrote:
> >
> > >  Hello All,
> > >
> > > I am trying to use petsc with Fortran. The initial mechanism to use
> petsc
> > > with FORTRAN77 that works fine. However, I found the Fortran 90
> interfaces
> > > difficult to work with. The very first issue is one has to recompile
> petsc
> > > with the Fortran compiler in case different to the one which was used
> to
> > > create the Fortran Mod files.
>
> Yes - one cannot mix fortran code compiled with 2 different compilers
> - or 2 different version of the same compiler wrt modules.
>
> But we generally recommend not mixing compilers   for C aswell as
> this
> can cause problems.
>
> Also you don't need to use modules to get f90 interfaces. You can include
> the f90 interface include files    directly in your code.
>
> #incude "finclude/petsc.h90"
>
> > > Also, we would still need to use C type
> > > #include statements along with Fortran 'use' statement which is not so
> > > optimal.
>
> Even if you were to use f90 modules #include is unavoidable as we still
> have other stuff in include files.
>
> check include/finclude/petscsysdef.h  include/finclude/petscsys.h
>
> > > Since Fortran 2003 offers great interface to C, did anyone try to
> write a
> > > F03 interface to petsc? Using F03 features, we can have very low data
> > > access to petsc database as well.
> > >
> >
> > F2003 support is fairly thin in the compiler world, and we have not had
> > much call for it. We do gratefully accept contributions.
>
> Currently we have the f90 interfaces [in include/finclude/ftn-auto/]
> automatically generated by bfort tool from the C interface. So to
> support F03 it would be ideal if there is such a tool. [or update
> bfort to do it]
>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/56033daa/attachment.html>

From knepley at gmail.com  Sun Oct 27 12:47:36 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 27 Oct 2013 12:47:36 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8iVYObVjj2h3H5P3PO7ndFcz-mWxTLUY8qkrTRWNnHW9A@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
	<CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
	<CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>
	<CAMYG4Gkh5a5Htu-LM=+2ztu7M58PK0bPGdCBeFY4jdFkW=Lq6Q@mail.gmail.com>
	<CAEhex8iVYObVjj2h3H5P3PO7ndFcz-mWxTLUY8qkrTRWNnHW9A@mail.gmail.com>
Message-ID: <CAMYG4G=vNukzcforn=Q8cvrBX53VL0Gp8F=+hCDeqM+NXigVcQ@mail.gmail.com>

On Sun, Oct 27, 2013 at 10:04 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:

>
>
>
> On Sun, Oct 27, 2013 at 2:53 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Sat, Oct 26, 2013 at 8:27 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>>
>>>
>>> On Sat, Oct 26, 2013 at 2:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>> case (of irregular domains):
>>>>>>>>>
>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>
>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>
>>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>>> takes care of the dirchlet values too.
>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>> the boundary points.
>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>>>> So is petscsection or dmplex suitable for this so that I can set
>>>>>>>>> up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>
>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>
>>>>>>>>
>>>>>>>> It sounds like the right solution for this is to use PetscSection
>>>>>>>> on top of DMDA. I am working on this, but it is really
>>>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>>>> we can help you.
>>>>>>>>
>>>>>>>
>>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>>> think.
>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>> the development version before.
>>>>>>>
>>>>>>
>>>>>> Okay,
>>>>>>
>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>
>>>>>
>>>>> I encountered errors when doing make on the 'next' branch. The errors
>>>>> are as follows (I tried attached the configure.log file but the email
>>>>> bounced back saying it awaits moderator approval for having too big an
>>>>> attachment, so I'm sending this one with only make.log attached. ) :
>>>>>
>>>>
>>>> They are fixed. Pull again and rebuild.
>>>>
>>>
>>> doing git pull in the next branch says "Already up-to-date.", I'm not
>>> sure if it should have said that then. Still tried ./configure and make
>>> again but returns the same error. Does it take sometime to get updated in
>>> the servers or do I need to do anything special again other than the
>>> following ? :
>>> git checkout next
>>> git pull
>>>
>>
>> Okay, it should be this
>>
>> git checkout next
>> git pull
>> make allfortranstubs
>> make
>>
>
> Thanks, but now it reports another error:
>
>          CXX arch-linux2-cxx-debug/obj/src/dm/impls/redundant/dmredundant.o
>          CXX
> arch-linux2-cxx-debug/obj/src/dm/impls/redundant/ftn-auto/dmredundantf.o
>          CXX arch-linux2-cxx-debug/obj/src/dm/impls/plex/plexcreate.o
> src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
> MatSetupDM_HYPREStruct(Mat, DM)?:
> src/dm/impls/da/hypre/mhyp.c:485:58: error: invalid conversion from
> ?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
> [-fpermissive]
> /home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27:
> error:   initializing argument 3 of ?PetscErrorCode
> DMDAGetGlobalIndices(DM, PetscInt*, const PetscInt**)? [-fpermissive]
> src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
> MatSetupDM_HYPRESStruct(Mat, DM)?:
> src/dm/impls/da/hypre/mhyp.c:963:58: error: invalid conversion from
> ?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
> [-fpermissive]
> /home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27:
> error:   initializing argument 3 of ?PetscErrorCode
> DMDAGetGlobalIndices(DM, PetscInt*, const PetscInt**)? [-fpermissive]
>

Yes, one of the test branched had a problem with Hypre. I pushed a fix.
Please try again.

  Thanks,

     Matt


> gmake[2]: *** [arch-linux2-cxx-debug/obj/src/dm/impls/da/hypre/mhyp.o]
> Error 1
> gmake[2]: *** Waiting for unfinished jobs....
> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
> gmake[1]: *** [gnumake] Error 2
> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
> **************************ERROR*************************************
>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>   Send it and arch-linux2-cxx-debug/conf/configure.log to
> petsc-maint at mcs.anl.gov
> ********************************************************************
> make: *** [all] Error 1
>
>
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>>>   Matt
>>>>
>>>>
>>>>>           CXX arch-linux2-cxx-debug/obj/src/
>>>>> mat/order/ftn-auto/spectralf.o
>>>>>          CXX
>>>>> arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
>>>>> src/mat/order/wbm.c: In function ?PetscErrorCode
>>>>> MatGetOrdering_WBM(Mat, MatOrderingType, _p_IS**, _p_IS**)?:
>>>>> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
>>>>> [-Wunused-but-set-variable]
>>>>> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
>>>>> [-Wunused-variable]
>>>>> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
>>>>> [-Wunused-but-set-variable]
>>>>> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
>>>>> [-Wunused-variable]
>>>>>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
>>>>> src/mat/order/ftn-auto/spectralf.c: In function ?void
>>>>> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
>>>>> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian?
>>>>> was not declared in this scope
>>>>> gmake[2]: ***
>>>>> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
>>>>> gmake[2]: *** Waiting for unfinished jobs....
>>>>> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
>>>>> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
>>>>> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
>>>>> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
>>>>> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not
>>>>> used [-Wunused-but-set-variable]
>>>>> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not
>>>>> used [-Wunused-but-set-variable]
>>>>> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not
>>>>> used [-Wunused-but-set-variable]
>>>>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>>> gmake[1]: *** [gnumake] Error 2
>>>>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>>> **************************ERROR*************************************
>>>>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>>>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>>>>> petsc-maint at mcs.anl.gov
>>>>> ********************************************************************
>>>>> make: *** [all] Error 1
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>> where you want them
>>>>>>
>>>>>> 3) Setup the solver as usual
>>>>>>
>>>>>> You can do 1) an 3) before we do 2).
>>>>>>
>>>>>>
>>>>>>  If not, just put the identity into
>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>> scalability or convergence.
>>>>>>>>
>>>>>>>
>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>> 1)  Having read Jed's answer here :
>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>
>>>>>>
>>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>>> is just decoupling these unknowns from the rest
>>>>>> of the problem.
>>>>>>
>>>>>>
>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>> rid of the identity rows too.
>>>>>>>
>>>>>>
>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>> conditioning worse.
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>> non-staggered case.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bishesh
>>>>>>>
>>>>>>>>
>>>>>>>>   Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bishesh
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/b58312c1/attachment-0001.html>

From stali at geology.wisc.edu  Sun Oct 27 14:51:01 2013
From: stali at geology.wisc.edu (Tabrez Ali)
Date: Sun, 27 Oct 2013 14:51:01 -0500
Subject: [petsc-users] Fortran interface -- low level access to petsc
 data
In-Reply-To: <CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
Message-ID: <526D6EA5.9090400@geology.wisc.edu>

On 10/27/2013 12:36 PM, Hossein Talebi wrote:
> Thank you for your answer.
>
> It is not true that F2003 support is that limited: Gfortran, Intel 
> Fortran, Cray Fortran, IBM Fortran and PGI Fortran all support nearly 
> full F2003. Some other ones support the C portability of F2003 
> although not supporting the rest.

You're 100% right but many systems often have older versions of these 
compilers that don't support the Fortran C interoperability. There is 
absolutely no excuse to not upgrade but then its not always possible.

Unless there is a significant gain in functionality (or ease of support) 
by moving to a Fortran 2003 compiler it would be better to be 
conservative. One advantage of being able to build PETSc on a old enough 
system (e.g., Debian Etch with GCC 4.1; glibc 2.4)  is that you can very 
easily build a static binary of your code and it will run pretty much on 
every Linux machine out there.

T

> * I can contribute on creating this interface. I have experience with 
> F2003 to C++ and it works fine while C is a lot easier. My idea is to 
> create the F03 interface to petsc as a separate tool. So, anyone 
> trying to use petsc with a F03 compiler can compile this interface 
> again to avoid the compiler inconsistency of Modules.
>
> We can work on a python (or similar) script to generate the 
> interfaces. For example, a project called gtk-fortran uses this 
> method. Can any of you guys help a little for planning the interface 
> design?
>
>
> Cheers
> Hossein
>
>
>
>
>
>
>
>
>
>
> On Sun, Oct 27, 2013 at 4:42 PM, Satish Balay <balay at mcs.anl.gov 
> <mailto:balay at mcs.anl.gov>> wrote:
>
>     On Sun, 27 Oct 2013, Matthew Knepley wrote:
>
>     > On Sun, Oct 27, 2013 at 6:16 AM, Hossein Talebi
>     <talebi.hossein at gmail.com <mailto:talebi.hossein at gmail.com>>wrote:
>     >
>     > >  Hello All,
>     > >
>     > > I am trying to use petsc with Fortran. The initial mechanism
>     to use petsc
>     > > with FORTRAN77 that works fine. However, I found the Fortran
>     90 interfaces
>     > > difficult to work with. The very first issue is one has to
>     recompile petsc
>     > > with the Fortran compiler in case different to the one which
>     was used to
>     > > create the Fortran Mod files.
>
>     Yes - one cannot mix fortran code compiled with 2 different compilers
>     - or 2 different version of the same compiler wrt modules.
>
>     But we generally recommend not mixing compilers   for C aswell as
>           this
>     can cause problems.
>
>     Also you don't need to use modules to get f90 interfaces. You can
>     include
>     the f90 interface include files    directly in your code.
>
>     #incude "finclude/petsc.h90"
>
>     > > Also, we would still need to use C type
>     > > #include statements along with Fortran 'use' statement which
>     is not so
>     > > optimal.
>
>     Even if you were to use f90 modules #include is unavoidable as we
>     still
>     have other stuff in include files.
>
>     check include/finclude/petscsysdef.h  include/finclude/petscsys.h
>
>     > > Since Fortran 2003 offers great interface to C, did anyone try
>     to write a
>     > > F03 interface to petsc? Using F03 features, we can have very
>     low data
>     > > access to petsc database as well.
>     > >
>     >
>     > F2003 support is fairly thin in the compiler world, and we have
>     not had
>     > much call for it. We do gratefully accept contributions.
>
>     Currently we have the f90 interfaces [in include/finclude/ftn-auto/]
>     automatically generated by bfort tool from the C interface. So to
>     support F03 it would be ideal if there is such a tool. [or update
>     bfort to do it]
>
>     Satish
>
>
>
>


-- 
No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131027/5f54e702/attachment.html>

From bisheshkh at gmail.com  Mon Oct 28 01:30:39 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Mon, 28 Oct 2013 07:30:39 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4G=vNukzcforn=Q8cvrBX53VL0Gp8F=+hCDeqM+NXigVcQ@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8gD_R8=26-R87PWCJOaZS0fr=qsQkRjyaXKUzSrNMScvA@mail.gmail.com>
	<CAMYG4G=TLEMnZrej0knxcXg-Bd1B3Pjj9TSmRkhsV551Vq8ezw@mail.gmail.com>
	<CAEhex8gW0Nz9j5FHASfd+PQU8X3QKgpaHfnwjdiBQTvcj=3rCA@mail.gmail.com>
	<CAMYG4Gkh5a5Htu-LM=+2ztu7M58PK0bPGdCBeFY4jdFkW=Lq6Q@mail.gmail.com>
	<CAEhex8iVYObVjj2h3H5P3PO7ndFcz-mWxTLUY8qkrTRWNnHW9A@mail.gmail.com>
	<CAMYG4G=vNukzcforn=Q8cvrBX53VL0Gp8F=+hCDeqM+NXigVcQ@mail.gmail.com>
Message-ID: <CAEhex8huLPyNf=-7s+nrGn7u0JEm0mVgHO1H9DLpL+pP4aKO1g@mail.gmail.com>

On Sun, Oct 27, 2013 at 6:47 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Oct 27, 2013 at 10:04 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Sun, Oct 27, 2013 at 2:53 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Sat, Oct 26, 2013 at 8:27 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Sat, Oct 26, 2013 at 2:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Sat, Oct 26, 2013 at 3:12 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Dear all,
>>>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>
>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>
>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>
>>>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>>>> takes care of the dirchlet values too.
>>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>>> the boundary points.
>>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>>> Later, I might have to extend this for the staggered grid case
>>>>>>>>>> too.
>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can set
>>>>>>>>>> up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>
>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It sounds like the right solution for this is to use PetscSection
>>>>>>>>> on top of DMDA. I am working on this, but it is really
>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>> development, we can help you.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>>>> think.
>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>>> the development version before.
>>>>>>>>
>>>>>>>
>>>>>>> Okay,
>>>>>>>
>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>
>>>>>>
>>>>>> I encountered errors when doing make on the 'next' branch. The errors
>>>>>> are as follows (I tried attached the configure.log file but the email
>>>>>> bounced back saying it awaits moderator approval for having too big an
>>>>>> attachment, so I'm sending this one with only make.log attached. ) :
>>>>>>
>>>>>
>>>>> They are fixed. Pull again and rebuild.
>>>>>
>>>>
>>>> doing git pull in the next branch says "Already up-to-date.", I'm not
>>>> sure if it should have said that then. Still tried ./configure and make
>>>> again but returns the same error. Does it take sometime to get updated in
>>>> the servers or do I need to do anything special again other than the
>>>> following ? :
>>>> git checkout next
>>>> git pull
>>>>
>>>
>>> Okay, it should be this
>>>
>>> git checkout next
>>> git pull
>>> make allfortranstubs
>>> make
>>>
>>
>> Thanks, but now it reports another error:
>>
>>          CXX
>> arch-linux2-cxx-debug/obj/src/dm/impls/redundant/dmredundant.o
>>          CXX
>> arch-linux2-cxx-debug/obj/src/dm/impls/redundant/ftn-auto/dmredundantf.o
>>          CXX arch-linux2-cxx-debug/obj/src/dm/impls/plex/plexcreate.o
>> src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
>> MatSetupDM_HYPREStruct(Mat, DM)?:
>> src/dm/impls/da/hypre/mhyp.c:485:58: error: invalid conversion from
>> ?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
>> [-fpermissive]
>> /home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27:
>> error:   initializing argument 3 of ?PetscErrorCode
>> DMDAGetGlobalIndices(DM, PetscInt*, const PetscInt**)? [-fpermissive]
>> src/dm/impls/da/hypre/mhyp.c: In function ?PetscErrorCode
>> MatSetupDM_HYPRESStruct(Mat, DM)?:
>> src/dm/impls/da/hypre/mhyp.c:963:58: error: invalid conversion from
>> ?PetscInt** {aka int**}? to ?const PetscInt** {aka const int**}?
>> [-fpermissive]
>> /home/bkhanal/Documents/softwares/petsc/include/petscdmda.h:68:27:
>> error:   initializing argument 3 of ?PetscErrorCode
>> DMDAGetGlobalIndices(DM, PetscInt*, const PetscInt**)? [-fpermissive]
>>
>
> Yes, one of the test branched had a problem with Hypre. I pushed a fix.
> Please try again.
>

Thanks, it works now. So (1) is done, I'm almost done with (3), will get
back soon.


>
>   Thanks,
>
>      Matt
>
>
>> gmake[2]: *** [arch-linux2-cxx-debug/obj/src/dm/impls/da/hypre/mhyp.o]
>> Error 1
>> gmake[2]: *** Waiting for unfinished jobs....
>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>> gmake[1]: *** [gnumake] Error 2
>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>> **************************ERROR*************************************
>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>> petsc-maint at mcs.anl.gov
>> ********************************************************************
>> make: *** [all] Error 1
>>
>>
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>>>   Matt
>>>>>
>>>>>
>>>>>>           CXX arch-linux2-cxx-debug/obj/src/
>>>>>> mat/order/ftn-auto/spectralf.o
>>>>>>          CXX
>>>>>> arch-linux2-cxx-debug/obj/src/mat/order/ftn-custom/zsorderf.o
>>>>>> src/mat/order/wbm.c: In function ?PetscErrorCode
>>>>>> MatGetOrdering_WBM(Mat, MatOrderingType, _p_IS**, _p_IS**)?:
>>>>>> src/mat/order/wbm.c:12:24: warning: variable ?cntl? set but not used
>>>>>> [-Wunused-but-set-variable]
>>>>>> src/mat/order/wbm.c:15:36: warning: unused variable ?num?
>>>>>> [-Wunused-variable]
>>>>>> src/mat/order/wbm.c:15:56: warning: variable ?icntl? set but not used
>>>>>> [-Wunused-but-set-variable]
>>>>>> src/mat/order/wbm.c:15:66: warning: unused variable ?info?
>>>>>> [-Wunused-variable]
>>>>>>          CXX arch-linux2-cxx-debug/obj/src/mat/matfd/fdmatrix.o
>>>>>> src/mat/order/ftn-auto/spectralf.c: In function ?void
>>>>>> matcreatelaplacian_(Mat, PetscReal*, PetscBool*, _p_Mat**, int*)?:
>>>>>> src/mat/order/ftn-auto/spectralf.c:44:40: error: ?MatCreateLaplacian?
>>>>>> was not declared in this scope
>>>>>> gmake[2]: ***
>>>>>> [arch-linux2-cxx-debug/obj/src/mat/order/ftn-auto/spectralf.o] Error 1
>>>>>> gmake[2]: *** Waiting for unfinished jobs....
>>>>>> src/mat/order/hslmc64.c: In function ?PetscErrorCode HSLmc64AD(const
>>>>>> PetscInt*, PetscInt*, PetscInt*, PetscInt*, const PetscInt*, const
>>>>>> PetscInt*, PetscScalar*, PetscInt*, PetscInt*, PetscInt*, PetscInt*,
>>>>>> PetscInt*, PetscScalar*, PetscInt*, PetscScalar*, PetscInt*)?:
>>>>>> src/mat/order/hslmc64.c:332:21: warning: variable ?warn1? set but not
>>>>>> used [-Wunused-but-set-variable]
>>>>>> src/mat/order/hslmc64.c:332:28: warning: variable ?warn2? set but not
>>>>>> used [-Wunused-but-set-variable]
>>>>>> src/mat/order/hslmc64.c:332:35: warning: variable ?warn4? set but not
>>>>>> used [-Wunused-but-set-variable]
>>>>>> gmake[2]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>>>> gmake[1]: *** [gnumake] Error 2
>>>>>> gmake[1]: Leaving directory `/home/bkhanal/Documents/softwares/petsc'
>>>>>> **************************ERROR*************************************
>>>>>>   Error during compile, check arch-linux2-cxx-debug/conf/make.log
>>>>>>   Send it and arch-linux2-cxx-debug/conf/configure.log to
>>>>>> petsc-maint at mcs.anl.gov
>>>>>> ********************************************************************
>>>>>> make: *** [all] Error 1
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>> where you want them
>>>>>>>
>>>>>>> 3) Setup the solver as usual
>>>>>>>
>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>
>>>>>>>
>>>>>>>  If not, just put the identity into
>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>> scalability or convergence.
>>>>>>>>>
>>>>>>>>
>>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>
>>>>>>>
>>>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>>>> is just decoupling these unknowns from the rest
>>>>>>> of the problem.
>>>>>>>
>>>>>>>
>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>> rid of the identity rows too.
>>>>>>>>
>>>>>>>
>>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>>> conditioning worse.
>>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>>> non-staggered case.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bishesh
>>>>>>>>
>>>>>>>>>
>>>>>>>>>   Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bishesh
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/f9edac80/attachment-0001.html>

From christophe.ortiz at ciemat.es  Mon Oct 28 03:47:13 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Mon, 28 Oct 2013 09:47:13 +0100
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <87txg53ymx.fsf@mcs.anl.gov>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
	<CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
	<87txg53ymx.fsf@mcs.anl.gov>
Message-ID: <CANBrw+5Rb5_MaEauJdz7ETczfUu-cf3zoB6nbuAs8FNvu3ZGwg@mail.gmail.com>

On Fri, Oct 25, 2013 at 10:04 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>
> > After playing around, I found something interesting, an option that I
> have
> > never seen in any example: TSSetEquationType. I did a TSGetEquationType
> and
> > I got TS_EQ_UNSPECIFIED. Then I thought, ok, let's try to specify the
> > equation type and I added the following line to the code
> >
> >   ierr = TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1);
>
> Thanks for tracking this down.  Indeed, that example enforces boundary
> conditions using an algebraic expression:
>
>     if (i == 0) {
>       f[i].u = hx * (x[i].u - user->uleft);
>       f[i].v = hx * (x[i].v - user->vleft);
>
> For this reason, we can only obtain the stage derivative by
> differencing, rather than by evaluating the function.
>
> We need to audit the examples and use TSSetEquationType everywhere that
> it's needed, which is probably in several of the examples.
>

Hope that it will help.

But just to understand what I'm doing, what does TSSetEquationType do
internally ?

For the type of problem I'm trying to solve, is TS_EQ_DAE_IMPLICIT_INDEX1
ok ? What is the difference with TS_EQ_DAE_IMPLICIT_INDEX2,3 or _INDEXHI ?

Christophe



>
> Note that Dirichlet boundary conditions can also be chosen by setting an
> appropriate initial condition and using the equation du/dt = 0.
>
> > Well...the results is totally different, and in my opinion much more
> > consistent. The value at the boundary is now the expected one:
> >
> > 0    10.000000     <-----
> > 1    9.888647
> > 2    9.777382
> > 3    9.666293
> > 4    9.555467
> >
> > ...
> > ...
> > 112    5.011530
> > 113    5.010665
> > 114    5.009880
> > 115    5.009169
> > ...
> > ...
> > 252    9.666293
> > 253    9.777382
> > 254    9.888647
> > 255    10.000000     <----
> >
> >
> > Moreover, I see much more effect of diffusion. Without this option,
> > diffusion is always very weak, whatever the diffusion coefficient one
> puts,
> > which always surprised me.
> >
> > What is the effect of the
> TSSetEquationType(ts,TS_EQ_DAE_IMPLICIT_INDEX1); ?
> >
> > Christophe
> >
> >
> > On Fri, Oct 25, 2013 at 11:47 AM, Christophe Ortiz <
> > christophe.ortiz at ciemat.es> wrote:
> >
> >> Hi guys,
> >>
> >> I was having some troubles with my code (dof >> 1) and I decided to go
> >> back to simple things. Therefore, I took ex25.c, which is a nice case
> for
> >> me: diffusion of 2 species and 1 reaction.
> >>
> >> First, to have a better mesh I set 256 points (instead of the 11) in
> >>
> >>   ierr =
> >> DMDACreate1d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,256,2,2,NULL,&da);
> >>
> >> Then, I did something simple. I removed reaction terms by commenting
> >>
> >> ierr = TSSetRHSFunction(ts,NULL,FormRHSFunction,&user);CHKERRQ(ierr);
> >>
> >> So we are left with 2 diffusion equations:
> >>
> >> u_t -alpha u_xx = 0
> >> v_t - alpha v_xx = 0
> >>
> >> Now, do something simple:
> >> In FormInitialSolution, for variable v, change it and simply set it to 5
> >> (for instance), ie, v will have uniform values from i=0 to i=mx-1.
> >>
> >> Now, for boundary conditions (Dirichlet here), set vleft and vright to
> 10,
> >> for instance.
> >>
> >> Since values at the boundaries (10.0) are higher than values of initial
> >> solution (5.0), what you expect is some diffusion towards the center
> from
> >> the boundaries. Since we use Dirichlet boundary conditions here, you
> expect
> >> the values of v at i=0 and i=mx-1 to be 10.0. It can't be something
> else.
> >> Well, that's not the case. I see that the value of v at i=0 and at
> i=mx-1
> >> is always the initial value (t=0). Here is a sample of the final values
> >> after diffusion:
> >>
> >> 0    5.000000
> >> 1    8.951153
> >> 2    8.414078
> >> 3    7.964193
> >> 4    7.581196
> >> ...
> >> ...
> >> 77    5.000235
> >> 78    5.000207
> >> 79    5.000182
> >> 80    5.000161
> >> ...
> >> ...
> >> 252    7.964193
> >> 253    8.414078
> >> 254    8.951153
> >> 255    5.000000
> >>
> >> Then I checked the value of v inside IFunction at each timestep. I found
> >> out that each time IFunction is called, the first value of v that is
> used
> >> to evaluate F is the initial value (5.0), instead of of the value set by
> >> Dirichlet (10). For next evaluations within the same timestep, it is
> 10.0,
> >> the value imposed by Dirichlet boundary conditions.
> >> Some output:
> >>
> >>       TSAdapt 'basic': step 324 accepted t=9.73302    + 9.496e-02
> >> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.569e-02
> >> 5.000000
> >> 10.000000
> >> 10.000000
> >> 10.000000
> >> 10.000000
> >>
> >>       TSAdapt 'basic': step 325 accepted t=9.82798    + 9.569e-02
> >> wlte=0.798 family='arkimex' scheme=0:'1bee' dt=9.642e-02
> >> 5.000000
> >> 10.000000
> >> 10.000000
> >> 10.000000
> >> 10.000000
> >>
> >> Is it a bug or did I miss something ? I would expect that to evaluate F
> >> within IFunction, it would always use the value set by Dirichlet.
> >>
> >>
> >> I use petsc 3.4.1and configured it with
> >>  --with-mpi=0 --with-fc=ifort --with-cc=icc --with-debugging=1
> >> --with-scalar-type=real --with-precision=double
> >> --with-blas-lapack-dir=/opt/intel/mkl.
> >>
> >> The options I used to run the example are
> >> -ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
> >> -ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
> >> -snes_linesearch_type bt.
> >>
> >>
> >> Christophe
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/1cc8e3a1/attachment.html>

From cedric.doucet at inria.fr  Mon Oct 28 05:25:25 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Mon, 28 Oct 2013 11:25:25 +0100 (CET)
Subject: [petsc-users] Partitioner in DMPlexDistribute
In-Reply-To: <CAMYG4G=GdmQAHBJAiDCM=_A_4y9zYtEg_ou-zrY6M=oHPmFebg@mail.gmail.com>
Message-ID: <487483245.83808.1382955925825.JavaMail.root@inria.fr>

Hello, 

I need to use DMPlexDistribute in a parallel finite element code. 
I would like to know which partitioner can be used with this function. 
I saw in an example that chaco is a possibility. 
I tried to use parmetis instead but I did not manage to do it. 
Furthermore, I do not see where the input partitioner is used in the source code of DMPlexDistribute. 

Best regards, 

C?dric Doucet 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/eefa65ef/attachment.html>

From juhaj at iki.fi  Mon Oct 28 05:27:07 2013
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Mon, 28 Oct 2013 11:27:07 +0100
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <87hac8jcgr.fsf@mcs.anl.gov>
References: <3178358.Nb05eOjZJN@dhcp071> <1665283.VQd2nzgDvj@rigel>
	<87hac8jcgr.fsf@mcs.anl.gov>
Message-ID: <3777330.qYkzf2BrSa@dhcp071>

> > For comparison, 3.4.2 gives (from the previous email): 354 MiB for 1
> > rank, 141 for 2 ranks and 81 for 4 ranks, which is a LOT less. I
> > suspect this might have something to do with the DA -> DMDA change?
> Hmm, I wonder where you're getting your numbers from.

>From /proc/self/stat, which (for me at least) gives exactly the same number as 
psutil.Process(os.getpid()).get_memory_info().rss (I suspect psutil actually 
reads /proc/self/stat as well).

Which brings up a point of your numbers

>   $ mpirun.hydra -np 1 python2 -c "$CMD"
>   8 MiB /
>   29 MiB
>   226 MiB

etc are way lower than mine. They are more or less in-line with my 3.2 tests, 
so something does not add up. My guess is "--with-64-bit-indices": if yours is 
with 32-bit indices, it's obvious you will get less memory consumption. Now, I 
need to apologise: my 3.2 test had 32-bit indices and all the old numbers I 
had for 3.3 did, tooo, but my 3.4.2 has 64-bit. I lately had to switch to 64-
bit indices and that just happened to coincide with the move to 3.4 series.

Perhaps this explains everything? (Of course, Barry's patch is still a good 
thing.)

Cheers,
-Juha
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 230 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/b2e2620a/attachment.pgp>

From bisheshkh at gmail.com  Mon Oct 28 05:30:34 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Mon, 28 Oct 2013 11:30:34 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
Message-ID: <CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>

On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> Dear all,
>>>> I would like to know if some of the petsc objects that I have not used
>>>> so far (IS, DMPlex, PetscSection) could be useful in the following case (of
>>>> irregular domains):
>>>>
>>>> Let's say that I have a 3D binary image (a cube).
>>>> The binary information of the image partitions the cube into a
>>>> computational domain and non-computational domain.
>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>> difference and say a dirichlet boundary condition
>>>>
>>>> I know that I can create a dmda that will let me access the information
>>>> from this 3D binary image, get all the coefficients, rhs values etc using
>>>> the natural indexing (i,j,k).
>>>>
>>>> Now, I would like to create a matrix corresponding to the laplace
>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>> takes care of the dirchlet values too.
>>>> But in this matrix it should have the rows corresponding to the nodes
>>>> only on the computational domain. It would be nice if I can easily (using
>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>> boundary points.
>>>> Then, once the system is solved, put the values of the solution back to
>>>> the corresponding positions in the binary image.
>>>> Later, I might have to extend this for the staggered grid case too.
>>>> So is petscsection or dmplex suitable for this so that I can set up the
>>>> matrix with something like DMCreateMatrix ? Or what would you suggest as a
>>>> suitable approach to this problem ?
>>>>
>>>> I have looked at the manual and that led me to search for a simpler
>>>> examples in petsc src directories. But most of the ones I encountered are
>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>> more as a distraction with FEM jargon!)
>>>>
>>>
>>> It sounds like the right solution for this is to use PetscSection on top
>>> of DMDA. I am working on this, but it is really
>>> alpha code. If you feel comfortable with that level of development, we
>>> can help you.
>>>
>>
>> Thanks, with the (short) experience of using Petsc so far and being
>> familiar with the awesomeness (quick and helpful replies) of this mailing
>> list, I would like to give it a try. Please give me some pointers to get
>> going for the example case I mentioned above. A simple example of using
>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>> think.
>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>> development version before.
>>
>
> Okay,
>
> 1)  clone the repository using Git and build the 'next' branch.
>
> 2) then we will need to create a PetscSection that puts unknowns where you
> want them
>
> 3) Setup the solver as usual
>
> You can do 1) an 3) before we do 2).
>
> I've done 1) and 3). I have one .cxx file that solves the system using
DMDA (putting identity into the rows corresponding to the cells that are
not used).
Please let me know what I should do now.



>
>  If not, just put the identity into
>>> the rows you do not use on the full cube. It will not hurt scalability
>>> or convergence.
>>>
>>
>> In the case of Poisson with Dirichlet condition this might be the case.
>> But is it always true that having identity rows in the system matrix will
>> not hurt convergence ? I thought otherwise for the following reasons:
>> 1)  Having read Jed's answer here :
>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>
>
> Jed is talking about a constraint on a the pressure at a point. This is
> just decoupling these unknowns from the rest
> of the problem.
>
>
>> 2) Some observation I am getting (but I am still doing more experiments
>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>> rid of using ghost cells method for the staggered grid and get rid of the
>> identity rows too.
>>
>
> It can change the exact iteration, but it does not make the matrix
> conditioning worse.
>
>    Matt
>
>
>>  Anyway please provide me with some pointers so that I can start trying
>> with petscsection on top of a dmda, in the beginning for non-staggered case.
>>
>> Thanks,
>> Bishesh
>>
>>>
>>>   Matt
>>>
>>>
>>>> Thanks,
>>>> Bishesh
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/1d8c1b69/attachment-0001.html>

From b.reinhardt at physik.uni-muenchen.de  Mon Oct 28 06:17:00 2013
From: b.reinhardt at physik.uni-muenchen.de (Bernhard Reinhardt)
Date: Mon, 28 Oct 2013 12:17:00 +0100
Subject: [petsc-users] DMPlex example with manual grid layout
Message-ID: <526E47AC.2050100@physik.uni-muenchen.de>

Hi there,

I am trying to familiarize myself with unstructured grids. For this I 
started with the example given in the petsc-manual (mesh of 2 triangles) 
and tried to modify it according to my needs.

However, the example seems to be incomplete. At least I was not able to 
create a matrix without applying a DMPlexSetDimension.

In the end I decided to set up an an example as simple as possible:
1D transport of a scalar in a bar with periodic boundaries.

Lets say I have a bar consisting of 3 volumes and 3 faces.
You could write this down as a vector of length 3 containing the scalar 
in the volumes and a 3x3 matrix containing the transport coefficients on 
the faces.

I have already tried many different ways of setting this up with plex 
but always failed. Attached you find the version that makes most sense 
to me. It gives me vectors of length 3 and a 3x3 matrix. However, the 
matrix contains no writable entries and I think the returned closures 
are not correct.

Is there a simple example with good documentation of the individual 
steps like in the manual which is complete and working?

Kind regards

Bernhard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.f90
Type: text/x-fortran
Size: 5122 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/76959009/attachment.bin>

From cedric.doucet at inria.fr  Mon Oct 28 06:17:22 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Mon, 28 Oct 2013 12:17:22 +0100 (CET)
Subject: [petsc-users] Partitioner in DMPlexDistribute
In-Reply-To: <487483245.83808.1382955925825.JavaMail.root@inria.fr>
Message-ID: <268529100.84710.1382959042850.JavaMail.root@inria.fr>

I was not looking at the latest version of the code (it seems online documentation has not been updated yet). 
Now, I can see that partitioner is used as an argument of DMPlexCreatePartition. 
However, there is a new argument: a PetscSF. 
I have two questions about this new implementation: 
1. which partitioner can be used? 
2. how to initialize the new PetscSF argument? 

----- Mail original -----

> De: "Cedric Doucet" <cedric.doucet at inria.fr>
> ?: petsc-users at mcs.anl.gov
> Envoy?: Lundi 28 Octobre 2013 11:25:25
> Objet: [petsc-users] Partitioner in DMPlexDistribute

> Hello,

> I need to use DMPlexDistribute in a parallel finite element code.
> I would like to know which partitioner can be used with this
> function.
> I saw in an example that chaco is a possibility.
> I tried to use parmetis instead but I did not manage to do it.
> Furthermore, I do not see where the input partitioner is used in the
> source code of DMPlexDistribute.

> Best regards,

> C?dric Doucet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/ed70d256/attachment.html>

From cedric.doucet at inria.fr  Mon Oct 28 07:05:41 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Mon, 28 Oct 2013 13:05:41 +0100 (CET)
Subject: [petsc-users] Partitioner in DMPlexDistribute
In-Reply-To: <268529100.84710.1382959042850.JavaMail.root@inria.fr>
Message-ID: <2015191383.85143.1382961941402.JavaMail.root@inria.fr>

I've just seen in DMPlexCreatePartition that only 'chaco' and 'metis' can be used. 

----- Mail original -----

> De: "Cedric Doucet" <cedric.doucet at inria.fr>
> ?: petsc-users at mcs.anl.gov
> Envoy?: Lundi 28 Octobre 2013 12:17:22
> Objet: Re: [petsc-users] Partitioner in DMPlexDistribute

> I was not looking at the latest version of the code (it seems online
> documentation has not been updated yet).
> Now, I can see that partitioner is used as an argument of
> DMPlexCreatePartition.
> However, there is a new argument: a PetscSF.
> I have two questions about this new implementation:
> 1. which partitioner can be used?
> 2. how to initialize the new PetscSF argument?

> ----- Mail original -----

> > De: "Cedric Doucet" <cedric.doucet at inria.fr>
> 
> > ?: petsc-users at mcs.anl.gov
> 
> > Envoy?: Lundi 28 Octobre 2013 11:25:25
> 
> > Objet: [petsc-users] Partitioner in DMPlexDistribute
> 

> > Hello,
> 

> > I need to use DMPlexDistribute in a parallel finite element code.
> 
> > I would like to know which partitioner can be used with this
> > function.
> 
> > I saw in an example that chaco is a possibility.
> 
> > I tried to use parmetis instead but I did not manage to do it.
> 
> > Furthermore, I do not see where the input partitioner is used in
> > the
> > source code of DMPlexDistribute.
> 

> > Best regards,
> 

> > C?dric Doucet
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/7c175707/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct 28 07:08:17 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 28 Oct 2013 07:08:17 -0500
Subject: [petsc-users] memory use of a DMDA
In-Reply-To: <3777330.qYkzf2BrSa@dhcp071>
References: <3178358.Nb05eOjZJN@dhcp071> <1665283.VQd2nzgDvj@rigel>
	<87hac8jcgr.fsf@mcs.anl.gov> <3777330.qYkzf2BrSa@dhcp071>
Message-ID: <5E99EAA2-4DCD-40B4-AD00-42EC96F0033B@mcs.anl.gov>


   The 64 bit integers would at most double the memory usage. It should actually be a bit less than doubling the memory usages since some of the memory allocated in DMSetUp_DA() are doubles whose size would remain unchanged.  Note that switching to 64 bit indices would not increase the Vec memory usage at all so the ratio of setup memory usage to a single vector memory usage would roughly double.

   Barry

On Oct 28, 2013, at 5:27 AM, Juha J?ykk? <juhaj at iki.fi> wrote:

>>> For comparison, 3.4.2 gives (from the previous email): 354 MiB for 1
>>> rank, 141 for 2 ranks and 81 for 4 ranks, which is a LOT less. I
>>> suspect this might have something to do with the DA -> DMDA change?
>> Hmm, I wonder where you're getting your numbers from.
> 
> From /proc/self/stat, which (for me at least) gives exactly the same number as 
> psutil.Process(os.getpid()).get_memory_info().rss (I suspect psutil actually 
> reads /proc/self/stat as well).
> 
> Which brings up a point of your numbers
> 
>>  $ mpirun.hydra -np 1 python2 -c "$CMD"
>>  8 MiB /
>>  29 MiB
>>  226 MiB
> 
> etc are way lower than mine. They are more or less in-line with my 3.2 tests, 
> so something does not add up. My guess is "--with-64-bit-indices": if yours is 
> with 32-bit indices, it's obvious you will get less memory consumption. Now, I 
> need to apologise: my 3.2 test had 32-bit indices and all the old numbers I 
> had for 3.3 did, tooo, but my 3.4.2 has 64-bit. I lately had to switch to 64-
> bit indices and that just happened to coincide with the move to 3.4 series.
> 
> Perhaps this explains everything? (Of course, Barry's patch is still a good 
> thing.)
> 
> Cheers,
> -Juha


From knepley at gmail.com  Mon Oct 28 07:40:30 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 07:40:30 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
Message-ID: <CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>

On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> Dear all,
>>>>> I would like to know if some of the petsc objects that I have not used
>>>>> so far (IS, DMPlex, PetscSection) could be useful in the following case (of
>>>>> irregular domains):
>>>>>
>>>>> Let's say that I have a 3D binary image (a cube).
>>>>> The binary information of the image partitions the cube into a
>>>>> computational domain and non-computational domain.
>>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>>> difference and say a dirichlet boundary condition
>>>>>
>>>>> I know that I can create a dmda that will let me access the
>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>> etc using the natural indexing (i,j,k).
>>>>>
>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>> takes care of the dirchlet values too.
>>>>> But in this matrix it should have the rows corresponding to the nodes
>>>>> only on the computational domain. It would be nice if I can easily (using
>>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>>> boundary points.
>>>>> Then, once the system is solved, put the values of the solution back
>>>>> to the corresponding positions in the binary image.
>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>> as a suitable approach to this problem ?
>>>>>
>>>>> I have looked at the manual and that led me to search for a simpler
>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>> more as a distraction with FEM jargon!)
>>>>>
>>>>
>>>> It sounds like the right solution for this is to use PetscSection on
>>>> top of DMDA. I am working on this, but it is really
>>>> alpha code. If you feel comfortable with that level of development, we
>>>> can help you.
>>>>
>>>
>>> Thanks, with the (short) experience of using Petsc so far and being
>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>> list, I would like to give it a try. Please give me some pointers to get
>>> going for the example case I mentioned above. A simple example of using
>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>> think.
>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>> development version before.
>>>
>>
>> Okay,
>>
>> 1)  clone the repository using Git and build the 'next' branch.
>>
>> 2) then we will need to create a PetscSection that puts unknowns where
>> you want them
>>
>> 3) Setup the solver as usual
>>
>> You can do 1) an 3) before we do 2).
>>
>> I've done 1) and 3). I have one .cxx file that solves the system using
> DMDA (putting identity into the rows corresponding to the cells that are
> not used).
> Please let me know what I should do now.
>

Okay, now write a loop to setup the PetscSection. I have the DMDA
partitioning cells, so you would have
unknowns in cells. The code should look like this:

PetscSectionCreate(comm, &s);
DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
PetscSectionSetChart(s, 0, nC);
for (k = zs; k < zs+zm; ++k) {
  for (j = ys; j < ys+ym; ++j) {
    for (i = xs; i < xs+xm; ++i) {
      PetscInt point;

      DMDAGetCellPoint(dm, i, j, k, &point);
      PetscSectionSetDof(s, point, dof); // You know how many dof are on
each vertex
    }
  }
}
PetscSectionSetUp(s);
DMSetDefaultSection(dm, s);
PetscSectionDestroy(&s);

I will merge the necessary stuff into 'next' to make this work.

  Thanks,

     Matt

>
>>  If not, just put the identity into
>>>> the rows you do not use on the full cube. It will not hurt scalability
>>>> or convergence.
>>>>
>>>
>>> In the case of Poisson with Dirichlet condition this might be the case.
>>> But is it always true that having identity rows in the system matrix will
>>> not hurt convergence ? I thought otherwise for the following reasons:
>>> 1)  Having read Jed's answer here :
>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>
>>
>> Jed is talking about a constraint on a the pressure at a point. This is
>> just decoupling these unknowns from the rest
>> of the problem.
>>
>>
>>> 2) Some observation I am getting (but I am still doing more experiments
>>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>>> rid of using ghost cells method for the staggered grid and get rid of the
>>> identity rows too.
>>>
>>
>> It can change the exact iteration, but it does not make the matrix
>> conditioning worse.
>>
>>    Matt
>>
>>
>>>  Anyway please provide me with some pointers so that I can start trying
>>> with petscsection on top of a dmda, in the beginning for non-staggered case.
>>>
>>> Thanks,
>>> Bishesh
>>>
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Bishesh
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/bb530099/attachment-0001.html>

From knepley at gmail.com  Mon Oct 28 07:42:59 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 07:42:59 -0500
Subject: [petsc-users] Partitioner in DMPlexDistribute
In-Reply-To: <268529100.84710.1382959042850.JavaMail.root@inria.fr>
References: <487483245.83808.1382955925825.JavaMail.root@inria.fr>
	<268529100.84710.1382959042850.JavaMail.root@inria.fr>
Message-ID: <CAMYG4Gmzc7KjYbbb1v9wUM3nqsrrR6zp4gV59MUQbGe1O6U=3w@mail.gmail.com>

On Mon, Oct 28, 2013 at 6:17 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:

>
> I was not looking at the latest version of the code (it seems online
> documentation has not been updated yet).
> Now, I can see that partitioner is used as an argument of
> DMPlexCreatePartition.
> However, there is a new argument: a PetscSF.
> I have two questions about this new implementation:
> 1. which partitioner can be used?
>

Yes, Chaco and Parmetis. These are really the only partitioners that
matter. If you want another one, it would not be
hard to put in, but the performance of all the others is not really
competitive with these.


> 2. how to initialize the new PetscSF argument?
>

You do not need to initialize it if you just call DMPlexDistribute().

  Matt


>
> ------------------------------
>
> *De: *"Cedric Doucet" <cedric.doucet at inria.fr>
> *?: *petsc-users at mcs.anl.gov
> *Envoy?: *Lundi 28 Octobre 2013 11:25:25
> *Objet: *[petsc-users] Partitioner in DMPlexDistribute
>
>
>
> Hello,
>
> I need to use DMPlexDistribute in a parallel finite element code.
> I would like to know which partitioner can be used with this function.
> I saw in an example that chaco is a possibility.
> I tried to use parmetis instead but I did not manage to do it.
> Furthermore, I do not see where the input partitioner is used in the
> source code of DMPlexDistribute.
>
> Best regards,
>
> C?dric Doucet
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/aa20e412/attachment.html>

From talebi.hossein at gmail.com  Mon Oct 28 07:50:46 2013
From: talebi.hossein at gmail.com (Hossein Talebi)
Date: Mon, 28 Oct 2013 13:50:46 +0100
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <526D6EA5.9090400@geology.wisc.edu>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
	<526D6EA5.9090400@geology.wisc.edu>
Message-ID: <CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>

Well, when dealing with Fortran I also often encounter software which
cannot be changed or upgraded to a more modern style.

What was in my mind is to create an interface additional to what is already
there in petsc. So this should be compiled along with anyone's code if the
used ocmpiler supports F03 or the necessary features of F03. This way the
older codes can be kept with the original F77 interface. Same thing they
did for Mpi and F03, F08 in the Mpi 3 standard if I'm not mistaken.





On Sun, Oct 27, 2013 at 8:51 PM, Tabrez Ali <stali at geology.wisc.edu> wrote:

>  On 10/27/2013 12:36 PM, Hossein Talebi wrote:
>
> Thank you for your answer.
>
>  It is not true that F2003 support is that limited: Gfortran, Intel
> Fortran, Cray Fortran, IBM Fortran and PGI Fortran all support nearly full
> F2003. Some other ones support the C portability of F2003 although not
> supporting the rest.
>
>
> You're 100% right but many systems often have older versions of these
> compilers that don't support the Fortran C interoperability. There is
> absolutely no excuse to not upgrade but then its not always possible.
>
> Unless there is a significant gain in functionality (or ease of support)
> by moving to a Fortran 2003 compiler it would be better to be conservative.
> One advantage of being able to build PETSc on a old enough system (e.g.,
> Debian Etch with GCC 4.1; glibc 2.4)  is that you can very easily build a
> static binary of your code and it will run pretty much on every Linux
> machine out there.
>
>
> T
>
>  * I can contribute on creating this interface. I have experience with
> F2003 to C++ and it works fine while C is a lot easier. My idea is to
> create the F03 interface to petsc as a separate tool. So, anyone trying to
> use petsc with a F03 compiler can compile this interface again to avoid the
> compiler inconsistency of Modules.
>
>  We can work on a python (or similar) script to generate the interfaces.
> For example, a project called gtk-fortran uses this method. Can any of you
> guys help a little for planning the interface design?
>
>
>  Cheers
> Hossein
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Oct 27, 2013 at 4:42 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> On Sun, 27 Oct 2013, Matthew Knepley wrote:
>>
>> > On Sun, Oct 27, 2013 at 6:16 AM, Hossein Talebi <
>> talebi.hossein at gmail.com>wrote:
>> >
>> > >  Hello All,
>> > >
>> > > I am trying to use petsc with Fortran. The initial mechanism to use
>> petsc
>> > > with FORTRAN77 that works fine. However, I found the Fortran 90
>> interfaces
>> > > difficult to work with. The very first issue is one has to recompile
>> petsc
>> > > with the Fortran compiler in case different to the one which was used
>> to
>> > > create the Fortran Mod files.
>>
>>  Yes - one cannot mix fortran code compiled with 2 different compilers
>> - or 2 different version of the same compiler wrt modules.
>>
>> But we generally recommend not mixing compilers   for C aswell as
>> this
>> can cause problems.
>>
>> Also you don't need to use modules to get f90 interfaces. You can include
>> the f90 interface include files    directly in your code.
>>
>> #incude "finclude/petsc.h90"
>>
>> > > Also, we would still need to use C type
>> > > #include statements along with Fortran 'use' statement which is not so
>> > > optimal.
>>
>>  Even if you were to use f90 modules #include is unavoidable as we still
>> have other stuff in include files.
>>
>> check include/finclude/petscsysdef.h  include/finclude/petscsys.h
>>
>> > > Since Fortran 2003 offers great interface to C, did anyone try to
>> write a
>> > > F03 interface to petsc? Using F03 features, we can have very low data
>> > > access to petsc database as well.
>> > >
>> >
>> > F2003 support is fairly thin in the compiler world, and we have not had
>> > much call for it. We do gratefully accept contributions.
>>
>>  Currently we have the f90 interfaces [in include/finclude/ftn-auto/]
>> automatically generated by bfort tool from the C interface. So to
>> support F03 it would be ideal if there is such a tool. [or update
>> bfort to do it]
>>
>> Satish
>>
>
>
>
>
>
> --
> No one trusts a model except the one who wrote it; Everyone trusts an observation except the one who made it- Harlow Shapley
>
>


-- 
Dr.-Ing. Hossein Talebi

Bauhaus-Universit?t Weimar
Graudate Research School 1462 "Evaluation of Coupled Numerical and
Experimental Partial Models in Structural Engineering "
Postdoctoral Researcher

Berkaer Str. 9
99425 Weimar
Tel.: +49/3643/58 41 03
Fax:  +49/3643/58 41 01
E-Mail: hossein.talebi at uni-weimar.de <maximilian.huber at uni-weimar.de>
www.uni-weimar.de/grk1462
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/87a52221/attachment.html>

From jedbrown at mcs.anl.gov  Mon Oct 28 08:15:13 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 07:15:13 -0600
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
	<526D6EA5.9090400@geology.wisc.edu>
	<CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>
Message-ID: <87y55d1qqm.fsf@mcs.anl.gov>

Hossein Talebi <talebi.hossein at gmail.com> writes:
> What was in my mind is to create an interface additional to what is already
> there in petsc. So this should be compiled along with anyone's code if the
> used ocmpiler supports F03 or the necessary features of F03. 

I don't think anyone will want to manually maintain the interface
definitions for all of PETSc.  (F2003 users can call the C interface
using the C types support now, but of course you'd rather have Fortran
interfaces so that you can have type checking.)  It seems to me that the
big win of F2003 is that it should be possible to automatically generate
almost all these definitions.  But how would you manage callbacks?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/75fb6cb6/attachment.pgp>

From talebi.hossein at gmail.com  Mon Oct 28 08:28:04 2013
From: talebi.hossein at gmail.com (Hossein Talebi)
Date: Mon, 28 Oct 2013 14:28:04 +0100
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <87y55d1qqm.fsf@mcs.anl.gov>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
	<526D6EA5.9090400@geology.wisc.edu>
	<CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>
	<87y55d1qqm.fsf@mcs.anl.gov>
Message-ID: <CAEjaKwDJEEJvk=hx-co_r2fbuaoyejQv1e-QvjX6wLc6r-v39g@mail.gmail.com>

Exactly, I personally look for:

1) type checking during the function calls. This helps also not looking
into the manual for every single petsc function call.

2) low level access to petsc data structure. For example in my own code I
already have my own hadnling of parallel vectors and matrices. When I use
petsc, I have to always copy in/out all the data which is not optimal. So,
it would be great if I can share or be change the petsc data in my Fortran
code i.e. not through the petsc function calls. I understand this might be
dangerous or not duable etc. But, I see no other solution.

3) easy definitions of petsc namespace through the Fortran modules, so I
can choose what to import (using the Fortran 'only' statement).

There is no general automatic interface generation for Fortran, as much as
I'm aware of. The gtk-fortran project, they wrote a python script to
generate this interface automatically and it really works nicely. I could
not imagine writiing such nice GTK GUI with Fortran.




On Mon, Oct 28, 2013 at 2:15 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Hossein Talebi <talebi.hossein at gmail.com> writes:
> > What was in my mind is to create an interface additional to what is
> already
> > there in petsc. So this should be compiled along with anyone's code if
> the
> > used ocmpiler supports F03 or the necessary features of F03.
>
> I don't think anyone will want to manually maintain the interface
> definitions for all of PETSc.  (F2003 users can call the C interface
> using the C types support now, but of course you'd rather have Fortran
> interfaces so that you can have type checking.)  It seems to me that the
> big win of F2003 is that it should be possible to automatically generate
> almost all these definitions.  But how would you manage callbacks?
>



-- 
Dr.-Ing. Hossein Talebi

Bauhaus-Universit?t Weimar
Graudate Research School 1462 "Evaluation of Coupled Numerical and
Experimental Partial Models in Structural Engineering "
Postdoctoral Researcher

Berkaer Str. 9
99425 Weimar
Tel.: +49/3643/58 41 03
Fax:  +49/3643/58 41 01
E-Mail: hossein.talebi at uni-weimar.de <maximilian.huber at uni-weimar.de>
www.uni-weimar.de/grk1462
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/b8c32b3e/attachment-0001.html>

From jedbrown at mcs.anl.gov  Mon Oct 28 09:00:56 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 08:00:56 -0600
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <CANBrw+5Rb5_MaEauJdz7ETczfUu-cf3zoB6nbuAs8FNvu3ZGwg@mail.gmail.com>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
	<CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
	<87txg53ymx.fsf@mcs.anl.gov>
	<CANBrw+5Rb5_MaEauJdz7ETczfUu-cf3zoB6nbuAs8FNvu3ZGwg@mail.gmail.com>
Message-ID: <87ppqp1omf.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> But just to understand what I'm doing, what does TSSetEquationType do
> internally ?

It tells the method what is necessary to evaluate stage derivatives, and
affects the starting method and (possibly) what can be carried over from
the last step.

> For the type of problem I'm trying to solve, is TS_EQ_DAE_IMPLICIT_INDEX1
> ok ? 

Yes.

> What is the difference with TS_EQ_DAE_IMPLICIT_INDEX2,3 or _INDEXHI ?

I believe these are currently unused, but if we add support for
computing consistent initial conditions, we'll have to know the index.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/32bc52d0/attachment.pgp>

From bisheshkh at gmail.com  Mon Oct 28 09:06:06 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Mon, 28 Oct 2013 15:06:06 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
Message-ID: <CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>

On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> Dear all,
>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>> case (of irregular domains):
>>>>>>
>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>> The binary information of the image partitions the cube into a
>>>>>> computational domain and non-computational domain.
>>>>>> I must solve a pde (say a Poisson equation) only on the computational
>>>>>> domains (e.g: two isolated spheres within the cube). I'm using finite
>>>>>> difference and say a dirichlet boundary condition
>>>>>>
>>>>>> I know that I can create a dmda that will let me access the
>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>> etc using the natural indexing (i,j,k).
>>>>>>
>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>> takes care of the dirchlet values too.
>>>>>> But in this matrix it should have the rows corresponding to the nodes
>>>>>> only on the computational domain. It would be nice if I can easily (using
>>>>>> (i,j,k) indexing) put on the rhs dirichlet values corresponding to the
>>>>>> boundary points.
>>>>>> Then, once the system is solved, put the values of the solution back
>>>>>> to the corresponding positions in the binary image.
>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>> as a suitable approach to this problem ?
>>>>>>
>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>> more as a distraction with FEM jargon!)
>>>>>>
>>>>>
>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>> top of DMDA. I am working on this, but it is really
>>>>> alpha code. If you feel comfortable with that level of development, we
>>>>> can help you.
>>>>>
>>>>
>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>> list, I would like to give it a try. Please give me some pointers to get
>>>> going for the example case I mentioned above. A simple example of using
>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>> think.
>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>> development version before.
>>>>
>>>
>>> Okay,
>>>
>>> 1)  clone the repository using Git and build the 'next' branch.
>>>
>>> 2) then we will need to create a PetscSection that puts unknowns where
>>> you want them
>>>
>>> 3) Setup the solver as usual
>>>
>>> You can do 1) an 3) before we do 2).
>>>
>>> I've done 1) and 3). I have one .cxx file that solves the system using
>> DMDA (putting identity into the rows corresponding to the cells that are
>> not used).
>> Please let me know what I should do now.
>>
>
> Okay, now write a loop to setup the PetscSection. I have the DMDA
> partitioning cells, so you would have
> unknowns in cells. The code should look like this:
>
> PetscSectionCreate(comm, &s);
> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
> PetscSectionSetChart(s, 0, nC);
> for (k = zs; k < zs+zm; ++k) {
>   for (j = ys; j < ys+ym; ++j) {
>     for (i = xs; i < xs+xm; ++i) {
>       PetscInt point;
>
>       DMDAGetCellPoint(dm, i, j, k, &point);
>       PetscSectionSetDof(s, point, dof); // You know how many dof are on
> each vertex
>     }
>   }
> }
> PetscSectionSetUp(s);
> DMSetDefaultSection(dm, s);
> PetscSectionDestroy(&s);
>
> I will merge the necessary stuff into 'next' to make this work.
>

I have put an example without the PetscSection here:
https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>From the code you have written above, how do we let PetscSection select
only those cells that lie in the computational domain ?  Is it that for
every "point" inside the above loop, we check whether it lies in the domain
or not, e.g using the function isPosInDomain(...) in the .cxx file I linked
to?



>   Thanks,
>
>      Matt
>
>>
>>>  If not, just put the identity into
>>>>> the rows you do not use on the full cube. It will not hurt scalability
>>>>> or convergence.
>>>>>
>>>>
>>>> In the case of Poisson with Dirichlet condition this might be the case.
>>>> But is it always true that having identity rows in the system matrix will
>>>> not hurt convergence ? I thought otherwise for the following reasons:
>>>> 1)  Having read Jed's answer here :
>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>
>>>
>>> Jed is talking about a constraint on a the pressure at a point. This is
>>> just decoupling these unknowns from the rest
>>> of the problem.
>>>
>>>
>>>> 2) Some observation I am getting (but I am still doing more experiments
>>>> to confirm) while solving my staggered-grid 3D stokes flow with schur
>>>> complement and using -pc_type gamg for A00 matrix. Putting the identity
>>>> rows for dirichlet boundaries and for ghost cells seemed to have effects on
>>>> its convergence. I'm hoping once I know how to use PetscSection, I can get
>>>> rid of using ghost cells method for the staggered grid and get rid of the
>>>> identity rows too.
>>>>
>>>
>>> It can change the exact iteration, but it does not make the matrix
>>> conditioning worse.
>>>
>>>    Matt
>>>
>>>
>>>>  Anyway please provide me with some pointers so that I can start
>>>> trying with petscsection on top of a dmda, in the beginning for
>>>> non-staggered case.
>>>>
>>>> Thanks,
>>>> Bishesh
>>>>
>>>>>
>>>>>   Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Bishesh
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/7a90cf45/attachment.html>

From lu_qin_2000 at yahoo.com  Mon Oct 28 09:36:31 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Mon, 28 Oct 2013 07:36:31 -0700 (PDT)
Subject: [petsc-users] Could not find make when building
	PETSc	in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
Message-ID: <1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>

Hi,
?
The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
?
I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
?
Thanks for?any hints,
Qin?
?
?



On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
  
pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.

It will go into 'maint' later..

Satish


On Thu, 24 Oct 2013, Satish Balay wrote:

> Perhaps we should add back the windows-python check?
> 
> [since we haven't yet added the windows python support]
> 
> Satish
> 
> On Thu, 24 Oct 2013, Qin Lu wrote:
> 
> > Chetan
> > 
> > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > 
> > Thanks a lot for all you guys'?helps.
> > 
> > Qin?
> > 
> > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > I vaguely recall seeing the "could not locate make" error
> > when a non-cygwin python is picked up first in path.
> > So ensure that "which python" points to cygwin python.
> > 
> > Another issue is that the --with-make argument, if it is
> > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > other drive letter in between where cygwin is installed).
> > 
> > Chetan
> > 
> > 
> > > -----Original Message-----
> > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > Sent: Thursday, October 24, 2013 8:43 AM
> > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > 
> > > Jed,
> > > 
> > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > in confugure, it gave the same error.
> > > 
> > > Thanks,
> > > Qin
> > > 
> > > 
> > > 
> > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > 
> > > > Hello,
> > > >
> > > > I am trying to build PETSc in Windows 7 with following steps:
> > > >
> > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > >
> > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > 
> > > You need to pass the full path to the make executable (or add it to your
> > > path), or you can use --download-make.
> > > 
> > > 
> > > > What should I do to let configure know the path of make?
> > > >
> > > > Many thanks for your help,
> > > >
> > > > Regard,
> > > > Qin
> > 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/548cd2ff/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error.log
Type: application/octet-stream
Size: 100348 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/548cd2ff/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 257901 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/548cd2ff/attachment-0003.obj>

From jedbrown at mcs.anl.gov  Mon Oct 28 09:45:09 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 08:45:09 -0600
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <CAEjaKwDJEEJvk=hx-co_r2fbuaoyejQv1e-QvjX6wLc6r-v39g@mail.gmail.com>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
	<526D6EA5.9090400@geology.wisc.edu>
	<CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>
	<87y55d1qqm.fsf@mcs.anl.gov>
	<CAEjaKwDJEEJvk=hx-co_r2fbuaoyejQv1e-QvjX6wLc6r-v39g@mail.gmail.com>
Message-ID: <87k3gx1mkq.fsf@mcs.anl.gov>

Hossein Talebi <talebi.hossein at gmail.com> writes:

> Exactly, I personally look for:
>
> 1) type checking during the function calls. This helps also not looking
> into the manual for every single petsc function call.
>
> 2) low level access to petsc data structure. For example in my own code I
> already have my own hadnling of parallel vectors and matrices. When I use
> petsc, I have to always copy in/out all the data which is not optimal. So,
> it would be great if I can share or be change the petsc data in my Fortran
> code i.e. not through the petsc function calls. I understand this might be
> dangerous or not duable etc. But, I see no other solution.

1. You really need to profile.  Copying a vector is cheap.

2. If you make your functions (to evaluate residuals, etc) take their
   inputs as explicit arguments, you can just pass the PETSc objects (or
   get the arrays from PETSc).

3. Parallel matrix assembly is almost certainly not something for which
   you want to hard-code the data structure.  PETSc provides generic
   interfaces for _incrementally_ assembling matrices, so that there is
   no extra storage and so the resulting data structure can be chosen at
   run-time.  I challenge you to find evidence that you can do this
   better.

4. If you really want to use _your_ storage for a Vec or Mat, you can
   use VecCreateMPIWithArray, MatCreateMPIAIJWithSplitArrays, and the
   like.

All of the above are available through _public_ interfaces that are
already available to Fortran callers.  Please don't use F2003 to expose
the _private_ struct definitions that we do not even offer to C callers.

> 3) easy definitions of petsc namespace through the Fortran modules, so I
> can choose what to import (using the Fortran 'only' statement).

PETSc uses prefixes for namespaces.  You can use "only" with the normal
modules, but I can't tell if you are suggesting changing the names to
avoid the prefix (which I don't think is a good idea and will lead to
some collisions).

> There is no general automatic interface generation for Fortran, as much as
> I'm aware of. The gtk-fortran project, they wrote a python script to
> generate this interface automatically and it really works nicely. I could
> not imagine writiing such nice GTK GUI with Fortran.

Hmm, it may be interesting to look at that script and see if it can be adapted.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/dbad11e3/attachment.pgp>

From knepley at gmail.com  Mon Oct 28 09:49:37 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 09:49:37 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
Message-ID: <CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>

On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>>
>>>
>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>> case (of irregular domains):
>>>>>>>
>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>> The binary information of the image partitions the cube into a
>>>>>>> computational domain and non-computational domain.
>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>
>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>
>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>> takes care of the dirchlet values too.
>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>> the boundary points.
>>>>>>> Then, once the system is solved, put the values of the solution back
>>>>>>> to the corresponding positions in the binary image.
>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>>> as a suitable approach to this problem ?
>>>>>>>
>>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>>> more as a distraction with FEM jargon!)
>>>>>>>
>>>>>>
>>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>>> top of DMDA. I am working on this, but it is really
>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>> we can help you.
>>>>>>
>>>>>
>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>> going for the example case I mentioned above. A simple example of using
>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>> think.
>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>>> development version before.
>>>>>
>>>>
>>>> Okay,
>>>>
>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>
>>>> 2) then we will need to create a PetscSection that puts unknowns where
>>>> you want them
>>>>
>>>> 3) Setup the solver as usual
>>>>
>>>> You can do 1) an 3) before we do 2).
>>>>
>>>> I've done 1) and 3). I have one .cxx file that solves the system using
>>> DMDA (putting identity into the rows corresponding to the cells that are
>>> not used).
>>> Please let me know what I should do now.
>>>
>>
>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>> partitioning cells, so you would have
>> unknowns in cells. The code should look like this:
>>
>> PetscSectionCreate(comm, &s);
>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>> PetscSectionSetChart(s, 0, nC);
>> for (k = zs; k < zs+zm; ++k) {
>>   for (j = ys; j < ys+ym; ++j) {
>>     for (i = xs; i < xs+xm; ++i) {
>>       PetscInt point;
>>
>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>       PetscSectionSetDof(s, point, dof); // You know how many dof are on
>> each vertex
>>     }
>>   }
>> }
>> PetscSectionSetUp(s);
>> DMSetDefaultSection(dm, s);
>> PetscSectionDestroy(&s);
>>
>> I will merge the necessary stuff into 'next' to make this work.
>>
>
> I have put an example without the PetscSection here:
> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
> From the code you have written above, how do we let PetscSection select
> only those cells that lie in the computational domain ?  Is it that for
> every "point" inside the above loop, we check whether it lies in the domain
> or not, e.g using the function isPosInDomain(...) in the .cxx file I linked
> to?
>

1) Give me permission to comment on the source (I am 'knepley')

2) You mask out the (i,j,k) that you do not want in that loop

   Matt


>
>>   Thanks,
>>
>>      Matt
>>
>>>
>>>>  If not, just put the identity into
>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>> scalability or convergence.
>>>>>>
>>>>>
>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>> case. But is it always true that having identity rows in the system matrix
>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>> 1)  Having read Jed's answer here :
>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>
>>>>
>>>> Jed is talking about a constraint on a the pressure at a point. This is
>>>> just decoupling these unknowns from the rest
>>>> of the problem.
>>>>
>>>>
>>>>> 2) Some observation I am getting (but I am still doing more
>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>> rid of the identity rows too.
>>>>>
>>>>
>>>> It can change the exact iteration, but it does not make the matrix
>>>> conditioning worse.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>>  Anyway please provide me with some pointers so that I can start
>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>> non-staggered case.
>>>>>
>>>>> Thanks,
>>>>> Bishesh
>>>>>
>>>>>>
>>>>>>   Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Bishesh
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/bacb2581/attachment.html>

From bisheshkh at gmail.com  Mon Oct 28 10:13:40 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Mon, 28 Oct 2013 16:13:40 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
Message-ID: <CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>

On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>> case (of irregular domains):
>>>>>>>>
>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>> computational domain and non-computational domain.
>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>
>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>
>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>> takes care of the dirchlet values too.
>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>> the boundary points.
>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>>> So is petscsection or dmplex suitable for this so that I can set up
>>>>>>>> the matrix with something like DMCreateMatrix ? Or what would you suggest
>>>>>>>> as a suitable approach to this problem ?
>>>>>>>>
>>>>>>>> I have looked at the manual and that led me to search for a simpler
>>>>>>>> examples in petsc src directories. But most of the ones I encountered are
>>>>>>>> with FEM (and I'm not familiar at all with FEM, so these examples serve
>>>>>>>> more as a distraction with FEM jargon!)
>>>>>>>>
>>>>>>>
>>>>>>> It sounds like the right solution for this is to use PetscSection on
>>>>>>> top of DMDA. I am working on this, but it is really
>>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>>> we can help you.
>>>>>>>
>>>>>>
>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>> think.
>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used the
>>>>>> development version before.
>>>>>>
>>>>>
>>>>> Okay,
>>>>>
>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>
>>>>> 2) then we will need to create a PetscSection that puts unknowns where
>>>>> you want them
>>>>>
>>>>> 3) Setup the solver as usual
>>>>>
>>>>> You can do 1) an 3) before we do 2).
>>>>>
>>>>> I've done 1) and 3). I have one .cxx file that solves the system using
>>>> DMDA (putting identity into the rows corresponding to the cells that are
>>>> not used).
>>>> Please let me know what I should do now.
>>>>
>>>
>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>> partitioning cells, so you would have
>>> unknowns in cells. The code should look like this:
>>>
>>> PetscSectionCreate(comm, &s);
>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>> PetscSectionSetChart(s, 0, nC);
>>> for (k = zs; k < zs+zm; ++k) {
>>>   for (j = ys; j < ys+ym; ++j) {
>>>     for (i = xs; i < xs+xm; ++i) {
>>>       PetscInt point;
>>>
>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>       PetscSectionSetDof(s, point, dof); // You know how many dof are on
>>> each vertex
>>>     }
>>>   }
>>> }
>>> PetscSectionSetUp(s);
>>> DMSetDefaultSection(dm, s);
>>> PetscSectionDestroy(&s);
>>>
>>> I will merge the necessary stuff into 'next' to make this work.
>>>
>>
>> I have put an example without the PetscSection here:
>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>> From the code you have written above, how do we let PetscSection select
>> only those cells that lie in the computational domain ?  Is it that for
>> every "point" inside the above loop, we check whether it lies in the domain
>> or not, e.g using the function isPosInDomain(...) in the .cxx file I linked
>> to?
>>
>
> 1) Give me permission to comment on the source (I am 'knepley')
>
> 2) You mask out the (i,j,k) that you do not want in that loop
>

Done.
I mask it out using isPosInDomain() function::
       if(isPosInDomain(&testPoisson,i,j,k)) {
            ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr);
            ierr = PetscSectionSetDof(s, point, testPoisson.mDof); // You
know how many dof are on each vertex
          }

And please let me know when I can rebuild the 'next' branch of petsc so
that DMDAGetCellPoint can be used. Currently compiler complains as it
cannot find it.

>
>    Matt
>
>
>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>>
>>>>>  If not, just put the identity into
>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>> scalability or convergence.
>>>>>>>
>>>>>>
>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>> 1)  Having read Jed's answer here :
>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>
>>>>>
>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>> is just decoupling these unknowns from the rest
>>>>> of the problem.
>>>>>
>>>>>
>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>> rid of the identity rows too.
>>>>>>
>>>>>
>>>>> It can change the exact iteration, but it does not make the matrix
>>>>> conditioning worse.
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>> non-staggered case.
>>>>>>
>>>>>> Thanks,
>>>>>> Bishesh
>>>>>>
>>>>>>>
>>>>>>>   Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bishesh
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/57d877c8/attachment-0001.html>

From knepley at gmail.com  Mon Oct 28 10:43:51 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 10:43:51 -0500
Subject: [petsc-users] DMPlex example with manual grid layout
In-Reply-To: <526E47AC.2050100@physik.uni-muenchen.de>
References: <526E47AC.2050100@physik.uni-muenchen.de>
Message-ID: <CAMYG4Gn8SML2f7c03p5Z5sST=L9+oh-JvhLEqemz3xj0VhHx2w@mail.gmail.com>

On Mon, Oct 28, 2013 at 6:17 AM, Bernhard Reinhardt <
b.reinhardt at physik.uni-muenchen.de> wrote:

> Hi there,
>
> I am trying to familiarize myself with unstructured grids. For this I
> started with the example given in the petsc-manual (mesh of 2 triangles)
> and tried to modify it according to my needs.
>
> However, the example seems to be incomplete. At least I was not able to
> create a matrix without applying a DMPlexSetDimension.
>
> In the end I decided to set up an an example as simple as possible:
> 1D transport of a scalar in a bar with periodic boundaries.
>
> Lets say I have a bar consisting of 3 volumes and 3 faces.
> You could write this down as a vector of length 3 containing the scalar in
> the volumes and a 3x3 matrix containing the transport coefficients on the
> faces.
>
> I have already tried many different ways of setting this up with plex but
> always failed. Attached you find the version that makes most sense to me.
> It gives me vectors of length 3 and a 3x3 matrix. However, the matrix
> contains no writable entries and I think the returned closures are not
> correct.
>
> Is there a simple example with good documentation of the individual steps
> like in the manual which is complete and working?
>

I have modified your example and it is attached. Here are a few points:

1) You ask for the closure of point 4 in your Vec, and get 1.0. This is
correct.

  The closure of 4 is {4, 1, 2}, the cell + its vertices. There are only
values on cells,
  so you get the value on 4, which is 1.0.

2) The default for preallocation is FEM-style topology, namely that
unknowns are connected to other
     unknowns with overlapping support. The support of your unknowns is
only the cell itself, so there
     are no connections by default. I put in a call to
DMPlexSetPreallocationCenterDimension(), which
     is the dimension of intermediate points. In FEM these are cells, but
in FVM they are vertices (or
     faces). I set your topology to FVM, and you should get the
preallocation you want.

  Thanks,

     Matt

Kind regards
>
> Bernhard
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/9f7be67d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.F90
Type: application/octet-stream
Size: 5122 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/9f7be67d/attachment.obj>

From talebi.hossein at gmail.com  Mon Oct 28 10:51:51 2013
From: talebi.hossein at gmail.com (Hossein Talebi)
Date: Mon, 28 Oct 2013 16:51:51 +0100
Subject: [petsc-users] Fortran interface -- low level access to petsc
	data
In-Reply-To: <87k3gx1mkq.fsf@mcs.anl.gov>
References: <CAEjaKwDhimWMiO1ZtKXJGRVPacPgW_5gCorGhwBH0N-39v9e-A@mail.gmail.com>
	<CAMYG4Gm7RFE+YNCqQ3xtZv3Swx2kbcQkt_r9EPY9qZ1ho0KJ+Q@mail.gmail.com>
	<alpine.LFD.2.03.1310271035581.3576@mcs.anl.gov>
	<CAEjaKwAPfF=x+Qqwf7JbGyZ_fiTpeHWDdVi-GVcwBRGf1o7hiQ@mail.gmail.com>
	<526D6EA5.9090400@geology.wisc.edu>
	<CAEjaKwCZPMsV5ybdrJ-1BNBw0e8UfxUUGEFNiUMiyR06917=Nw@mail.gmail.com>
	<87y55d1qqm.fsf@mcs.anl.gov>
	<CAEjaKwDJEEJvk=hx-co_r2fbuaoyejQv1e-QvjX6wLc6r-v39g@mail.gmail.com>
	<87k3gx1mkq.fsf@mcs.anl.gov>
Message-ID: <CAEjaKwDBLokDeUzn0gLeNdEe310JmD_g=9Zf=LuMdVJNCJRgJg@mail.gmail.com>

1. You really need to profile.  Copying a vector is cheap.

>>>You are right. Maybe I make a big thing for this copy in/out and it is
not that problematic or slow even for the whole 2D matrix. I have to test
it with a large one to see.



> 2. If you make your functions (to evaluate residuals, etc) take their
>    inputs as explicit arguments, you can just pass the PETSc objects (or

   get the arrays from PETSc).
>

>>>I did not understand this completely.

>
> 3. Parallel matrix assembly is almost certainly not something for which
>    you want to hard-code the data structure.  PETSc provides generic
>    interfaces for _incrementally_ assembling matrices, so that there is
>    no extra storage and so the resulting data structure can be chosen at
>    run-time.  I challenge you to find evidence that you can do this
>    better.
>
> >>>Well, no wonder petsc is a very general and mature solution. I have
coded this already and it works fine with couple of other solvers. I cannot
say it is faster/better than petsc and I even don't think the comparison
would be correct. Though I would accept the challenge if I had time :-). I
have the single code/single language strategy so my code should be able to
compile and run without any external library. I attach many external
libraries for additional tasks. Anyhow, my primary aim is to use petsc for
its nice linear equation solvers.



> 4. If you really want to use _your_ storage for a Vec or Mat, you can
>    use VecCreateMPIWithArray, MatCreateMPIAIJWithSplitArrays, and the
>    like.
>
This looks to be a solution. Thank you.



> All of the above are available through _public_ interfaces that are
> already available to Fortran callers.  Please don't use F2003 to expose
> the _private_ struct definitions that we do not even offer to C callers.
>
> >>>I agree.


> > 3) easy definitions of petsc namespace through the Fortran modules, so I
> > can choose what to import (using the Fortran 'only' statement).
>
> PETSc uses prefixes for namespaces.  You can use "only" with the normal
> modules, but I can't tell if you are suggesting changing the names to
> avoid the prefix (which I don't think is a good idea and will lead to
> some collisions).
>
>>>Not to change the namespace but to limit it. When I put C-type include
statements it brings all mostly unnecessary namespace which might collide
with others, as you mentioned.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/bf869965/attachment.html>

From balay at mcs.anl.gov  Mon Oct 28 11:16:52 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 28 Oct 2013 11:16:52 -0500 (CDT)
Subject: [petsc-users] Could not find make when
 building	PETSc	in	Windows 7
In-Reply-To: <1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>

> /cygdrive/c/Windows/SUA/common/rm -f -rf /cygdrive/c/cygwin_cache/petsc-3.4.3/arch-mswin-c-opt/lib/libpetsc*.*

You appear to have non cygwin stuff in your PATH. Can you remove these
things from your PATH - make sure grep in PATH is from cygwin - and
rerun configure && make

Satish


On Mon, 28 Oct 2013, Qin Lu wrote:

> Hi,
> ?
> The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
> ?
> I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
> ?
> Thanks for?any hints,
> Qin?
> ?
> ?
> 
> 
> 
> On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>   
> pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.
> 
> It will go into 'maint' later..
> 
> Satish
> 
> 
> On Thu, 24 Oct 2013, Satish Balay wrote:
> 
> > Perhaps we should add back the windows-python check?
> > 
> > [since we haven't yet added the windows python support]
> > 
> > Satish
> > 
> > On Thu, 24 Oct 2013, Qin Lu wrote:
> > 
> > > Chetan
> > > 
> > > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > > 
> > > Thanks a lot for all you guys'?helps.
> > > 
> > > Qin?
> > > 
> > > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > > I vaguely recall seeing the "could not locate make" error
> > > when a non-cygwin python is picked up first in path.
> > > So ensure that "which python" points to cygwin python.
> > > 
> > > Another issue is that the --with-make argument, if it is
> > > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > > other drive letter in between where cygwin is installed).
> > > 
> > > Chetan
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > > Sent: Thursday, October 24, 2013 8:43 AM
> > > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > > 
> > > > Jed,
> > > > 
> > > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > > in confugure, it gave the same error.
> > > > 
> > > > Thanks,
> > > > Qin
> > > > 
> > > > 
> > > > 
> > > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > > 
> > > > > Hello,
> > > > >
> > > > > I am trying to build PETSc in Windows 7 with following steps:
> > > > >
> > > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > > >
> > > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > > 
> > > > You need to pass the full path to the make executable (or add it to your
> > > > path), or you can use --download-make.
> > > > 
> > > > 
> > > > > What should I do to let configure know the path of make?
> > > > >
> > > > > Many thanks for your help,
> > > > >
> > > > > Regard,
> > > > > Qin
> > > 
> >

From lu_qin_2000 at yahoo.com  Mon Oct 28 11:39:53 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Mon, 28 Oct 2013 09:39:53 -0700 (PDT)
Subject: [petsc-users] Could not find make when
	building	PETSc	in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
Message-ID: <1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>

Satish,
?
As far as I know, cygwin reads the Windows enviromental variable PATH. Can I not change the the setting in in the Windows enviromental variables (since other programs need them), but change the PATH used by cygwin only? Which file should I change?
?
Thanks,
Qin



On Monday, October 28, 2013 11:16 AM, Satish Balay <balay at mcs.anl.gov> wrote:
  
> /cygdrive/c/Windows/SUA/common/rm -f -rf /cygdrive/c/cygwin_cache/petsc-3.4.3/arch-mswin-c-opt/lib/libpetsc*.*

You appear to have non cygwin stuff in your PATH. Can you remove these
things from your PATH - make sure grep in PATH is from cygwin - and
rerun configure && make

Satish



On Mon, 28 Oct 2013, Qin Lu wrote:

> Hi,
> ?
> The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
> ?
> I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
> ?
> Thanks for?any hints,
> Qin?
> ?
> ?
> 
> 
> 
> On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>? 
> pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.
> 
> It will go into 'maint' later..
> 
> Satish
> 
> 
> On Thu, 24 Oct 2013, Satish Balay wrote:
> 
> > Perhaps we should add back the windows-python check?
> > 
> > [since we haven't yet added the windows python support]
> > 
> > Satish
> > 
> > On Thu, 24 Oct 2013, Qin Lu wrote:
> > 
> > > Chetan
> > > 
> > > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > > 
> > > Thanks a lot for all you guys'?helps.
> > > 
> > > Qin?
> > > 
> > > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > > I vaguely recall seeing the "could not locate make" error
> > > when a non-cygwin python is picked up first in path.
> > > So ensure that "which python" points to cygwin python.
> > > 
> > > Another issue is that the --with-make argument, if it is
> > > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > > other drive letter in between where cygwin is installed).
> > > 
> > > Chetan
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > > Sent: Thursday, October 24, 2013 8:43 AM
> > > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > > 
> > > > Jed,
> > > > 
> > > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > > in confugure, it gave the same error.
> > > > 
> > > > Thanks,
> > > > Qin
> > > > 
> > > > 
> > > > 
> > > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > > 
> > > > > Hello,
> > > > >
> > > > > I am trying to build PETSc in Windows 7 with following steps:
> > > > >
> > > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > > >
> > > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > > 
> > > > You need to pass the full path to the make executable (or add it to your
> > > > path), or you can use --download-make.
> > > > 
> > > > 
> > > > > What should I do to let configure know the path of make?
> > > > >
> > > > > Many thanks for your help,
> > > > >
> > > > > Regard,
> > > > > Qin
> > > 
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/fd1745ea/attachment-0001.html>

From balay at mcs.anl.gov  Mon Oct 28 11:50:05 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 28 Oct 2013 11:50:05 -0500 (CDT)
Subject: [petsc-users] Could not find make
 when	building	PETSc	in	Windows 7
In-Reply-To: <1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>

cygwin should have already setup PATH correctly so that its listed first

What do you have for:

echo $PATH
which make
which rm
which grep
export PATH=/usr/bin:$PATH
which make
which rm
which grep

Satish


On Mon, 28 Oct 2013, Qin Lu wrote:

> Satish,
> ?
> As far as I know, cygwin reads the Windows enviromental variable PATH. Can I not change the the setting in in the Windows enviromental variables (since other programs need them), but change the PATH used by cygwin only? Which file should I change?
> ?
> Thanks,
> Qin
> 
> 
> 
> On Monday, October 28, 2013 11:16 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>   
> > /cygdrive/c/Windows/SUA/common/rm -f -rf /cygdrive/c/cygwin_cache/petsc-3.4.3/arch-mswin-c-opt/lib/libpetsc*.*
> 
> You appear to have non cygwin stuff in your PATH. Can you remove these
> things from your PATH - make sure grep in PATH is from cygwin - and
> rerun configure && make
> 
> Satish
> 
> 
> 
> On Mon, 28 Oct 2013, Qin Lu wrote:
> 
> > Hi,
> > ?
> > The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
> > ?
> > I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
> > ?
> > Thanks for?any hints,
> > Qin?
> > ?
> > ?
> > 
> > 
> > 
> > On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >? 
> > pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.
> > 
> > It will go into 'maint' later..
> > 
> > Satish
> > 
> > 
> > On Thu, 24 Oct 2013, Satish Balay wrote:
> > 
> > > Perhaps we should add back the windows-python check?
> > > 
> > > [since we haven't yet added the windows python support]
> > > 
> > > Satish
> > > 
> > > On Thu, 24 Oct 2013, Qin Lu wrote:
> > > 
> > > > Chetan
> > > > 
> > > > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > > > 
> > > > Thanks a lot for all you guys'?helps.
> > > > 
> > > > Qin?
> > > > 
> > > > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > > > I vaguely recall seeing the "could not locate make" error
> > > > when a non-cygwin python is picked up first in path.
> > > > So ensure that "which python" points to cygwin python.
> > > > 
> > > > Another issue is that the --with-make argument, if it is
> > > > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > > > other drive letter in between where cygwin is installed).
> > > > 
> > > > Chetan
> > > > 
> > > > 
> > > > > -----Original Message-----
> > > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > > > Sent: Thursday, October 24, 2013 8:43 AM
> > > > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > > > 
> > > > > Jed,
> > > > > 
> > > > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > > > in confugure, it gave the same error.
> > > > > 
> > > > > Thanks,
> > > > > Qin
> > > > > 
> > > > > 
> > > > > 
> > > > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > > > 
> > > > > > Hello,
> > > > > >
> > > > > > I am trying to build PETSc in Windows 7 with following steps:
> > > > > >
> > > > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > > > >
> > > > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > > > 
> > > > > You need to pass the full path to the make executable (or add it to your
> > > > > path), or you can use --download-make.
> > > > > 
> > > > > 
> > > > > > What should I do to let configure know the path of make?
> > > > > >
> > > > > > Many thanks for your help,
> > > > > >
> > > > > > Regard,
> > > > > > Qin
> > > > 
> > >

From knepley at gmail.com  Mon Oct 28 12:19:34 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 12:19:34 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
Message-ID: <CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>

On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:

>
>
>
> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>> case (of irregular domains):
>>>>>>>>>
>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>
>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>
>>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>>> takes care of the dirchlet values too.
>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>> the boundary points.
>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>> Later, I might have to extend this for the staggered grid case too.
>>>>>>>>> So is petscsection or dmplex suitable for this so that I can set
>>>>>>>>> up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>
>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>
>>>>>>>>
>>>>>>>> It sounds like the right solution for this is to use PetscSection
>>>>>>>> on top of DMDA. I am working on this, but it is really
>>>>>>>> alpha code. If you feel comfortable with that level of development,
>>>>>>>> we can help you.
>>>>>>>>
>>>>>>>
>>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>>> think.
>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>> the development version before.
>>>>>>>
>>>>>>
>>>>>> Okay,
>>>>>>
>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>
>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>> where you want them
>>>>>>
>>>>>> 3) Setup the solver as usual
>>>>>>
>>>>>> You can do 1) an 3) before we do 2).
>>>>>>
>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>> are not used).
>>>>> Please let me know what I should do now.
>>>>>
>>>>
>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>> partitioning cells, so you would have
>>>> unknowns in cells. The code should look like this:
>>>>
>>>> PetscSectionCreate(comm, &s);
>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>> PetscSectionSetChart(s, 0, nC);
>>>> for (k = zs; k < zs+zm; ++k) {
>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>       PetscInt point;
>>>>
>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof are
>>>> on each vertex
>>>>     }
>>>>   }
>>>> }
>>>> PetscSectionSetUp(s);
>>>> DMSetDefaultSection(dm, s);
>>>> PetscSectionDestroy(&s);
>>>>
>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>
>>>
>>> I have put an example without the PetscSection here:
>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>> From the code you have written above, how do we let PetscSection select
>>> only those cells that lie in the computational domain ?  Is it that for
>>> every "point" inside the above loop, we check whether it lies in the domain
>>> or not, e.g using the function isPosInDomain(...) in the .cxx file I linked
>>> to?
>>>
>>
>> 1) Give me permission to comment on the source (I am 'knepley')
>>
>> 2) You mask out the (i,j,k) that you do not want in that loop
>>
>
> Done.
> I mask it out using isPosInDomain() function::
>        if(isPosInDomain(&testPoisson,i,j,k)) {
>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr);
>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); // You
> know how many dof are on each vertex
>           }
>
> And please let me know when I can rebuild the 'next' branch of petsc so
> that DMDAGetCellPoint can be used. Currently compiler complains as it
> cannot find it.
>

Pushed.

   Matt


>
>>    Matt
>>
>>
>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>>
>>>>>>  If not, just put the identity into
>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>> scalability or convergence.
>>>>>>>>
>>>>>>>
>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>> 1)  Having read Jed's answer here :
>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>
>>>>>>
>>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>>> is just decoupling these unknowns from the rest
>>>>>> of the problem.
>>>>>>
>>>>>>
>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>> rid of the identity rows too.
>>>>>>>
>>>>>>
>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>> conditioning worse.
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>> non-staggered case.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bishesh
>>>>>>>
>>>>>>>>
>>>>>>>>   Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bishesh
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/9497d426/attachment.html>

From shiyuangu at gmail.com  Mon Oct 28 13:15:55 2013
From: shiyuangu at gmail.com (Shiyuan Gu)
Date: Mon, 28 Oct 2013 13:15:55 -0500
Subject: [petsc-users] viewer support for numpy
Message-ID: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>

Hi all,
     Does Petsc provide any support to export and view a petsc vector/mat ?
Thanks.

Shiyuan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/1d9c92cc/attachment.html>

From jedbrown at mcs.anl.gov  Mon Oct 28 13:18:35 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 12:18:35 -0600
Subject: [petsc-users] viewer support for numpy
In-Reply-To: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
References: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
Message-ID: <8761shz2bo.fsf@mcs.anl.gov>

Shiyuan Gu <shiyuangu at gmail.com> writes:

> Hi all,
>      Does Petsc provide any support to export and view a petsc vector/mat ?

See $PETSC_DIR/bin/pythonscripts/PetscBinaryIO.py.  You can read or
write the PETSc format as Numpy matrices.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/fc6bd02b/attachment-0001.pgp>

From shiyuangu at gmail.com  Mon Oct 28 13:56:04 2013
From: shiyuangu at gmail.com (Shiyuan Gu)
Date: Mon, 28 Oct 2013 13:56:04 -0500
Subject: [petsc-users] viewer support for numpy
In-Reply-To: <8761shz2bo.fsf@mcs.anl.gov>
References: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
	<8761shz2bo.fsf@mcs.anl.gov>
Message-ID: <CAEeiV_UTa+VyZijzdRox9gKEJSRhy48Wg=MOBF7TB3usdpkbBw@mail.gmail.com>

Thanks Jed.
Is there an convenient way to save petsc objects to a binary file in gdb?




On Mon, Oct 28, 2013 at 1:18 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Shiyuan Gu <shiyuangu at gmail.com> writes:
>
> > Hi all,
> >      Does Petsc provide any support to export and view a petsc
> vector/mat ?
>
> See $PETSC_DIR/bin/pythonscripts/PetscBinaryIO.py.  You can read or
> write the PETSc format as Numpy matrices.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/072e1ce2/attachment.html>

From jedbrown at mcs.anl.gov  Mon Oct 28 13:57:52 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 12:57:52 -0600
Subject: [petsc-users] viewer support for numpy
In-Reply-To: <CAEeiV_UTa+VyZijzdRox9gKEJSRhy48Wg=MOBF7TB3usdpkbBw@mail.gmail.com>
References: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
	<8761shz2bo.fsf@mcs.anl.gov>
	<CAEeiV_UTa+VyZijzdRox9gKEJSRhy48Wg=MOBF7TB3usdpkbBw@mail.gmail.com>
Message-ID: <87wqkxxlxr.fsf@mcs.anl.gov>

Shiyuan Gu <shiyuangu at gmail.com> writes:

> Thanks Jed.
> Is there an convenient way to save petsc objects to a binary file in gdb?

call VecView(X,PETSC_VIEWER_BINARY_WORLD)?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/68c7c94d/attachment.pgp>

From knepley at gmail.com  Mon Oct 28 14:13:43 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 14:13:43 -0500
Subject: [petsc-users] viewer support for numpy
In-Reply-To: <87wqkxxlxr.fsf@mcs.anl.gov>
References: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
	<8761shz2bo.fsf@mcs.anl.gov>
	<CAEeiV_UTa+VyZijzdRox9gKEJSRhy48Wg=MOBF7TB3usdpkbBw@mail.gmail.com>
	<87wqkxxlxr.fsf@mcs.anl.gov>
Message-ID: <CAMYG4Gkw8rCufJ2oYE8rRNmxFZJt7XEmFqcNfq2oNz1R6AuBLQ@mail.gmail.com>

On Mon, Oct 28, 2013 at 1:57 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Shiyuan Gu <shiyuangu at gmail.com> writes:
>
> > Thanks Jed.
> > Is there an convenient way to save petsc objects to a binary file in gdb?
>
> call VecView(X,PETSC_VIEWER_BINARY_WORLD)?
>

That never works for me. I use

  call VecView(X, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD))

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/ab06d300/attachment.html>

From jedbrown at mcs.anl.gov  Mon Oct 28 14:16:59 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 Oct 2013 13:16:59 -0600
Subject: [petsc-users] viewer support for numpy
In-Reply-To: <CAMYG4Gkw8rCufJ2oYE8rRNmxFZJt7XEmFqcNfq2oNz1R6AuBLQ@mail.gmail.com>
References: <CAEeiV_Wz6aPd8UOQmxL4G2x+RO4fX2RBQuLp5pyXyZ5WT+TVdw@mail.gmail.com>
	<8761shz2bo.fsf@mcs.anl.gov>
	<CAEeiV_UTa+VyZijzdRox9gKEJSRhy48Wg=MOBF7TB3usdpkbBw@mail.gmail.com>
	<87wqkxxlxr.fsf@mcs.anl.gov>
	<CAMYG4Gkw8rCufJ2oYE8rRNmxFZJt7XEmFqcNfq2oNz1R6AuBLQ@mail.gmail.com>
Message-ID: <87eh75xl1w.fsf@mcs.anl.gov>

Matthew Knepley <knepley at gmail.com> writes:
> That never works for me. I use
>
>   call VecView(X, PETSC_VIEWER_BINARY_(PETSC_COMM_WORLD))

Oh, right.  The debugger doesn't see macros to expand them automatically.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/0636b33a/attachment.pgp>

From bisheshkh at gmail.com  Mon Oct 28 14:51:30 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Mon, 28 Oct 2013 20:51:30 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
Message-ID: <CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>

On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Dear all,
>>>>>>>>>> I would like to know if some of the petsc objects that I have not
>>>>>>>>>> used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>
>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>
>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>
>>>>>>>>>> Now, I would like to create a matrix corresponding to the laplace
>>>>>>>>>> operator (e.g. with standard 7 pt. stencil), and the corresponding RHS that
>>>>>>>>>> takes care of the dirchlet values too.
>>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>>> the boundary points.
>>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>>> Later, I might have to extend this for the staggered grid case
>>>>>>>>>> too.
>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can set
>>>>>>>>>> up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>
>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It sounds like the right solution for this is to use PetscSection
>>>>>>>>> on top of DMDA. I am working on this, but it is really
>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>> development, we can help you.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks, with the (short) experience of using Petsc so far and being
>>>>>>>> familiar with the awesomeness (quick and helpful replies) of this mailing
>>>>>>>> list, I would like to give it a try. Please give me some pointers to get
>>>>>>>> going for the example case I mentioned above. A simple example of using
>>>>>>>> PetscSection along with DMDA for finite volume (No FEM) would be great I
>>>>>>>> think.
>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>>> the development version before.
>>>>>>>>
>>>>>>>
>>>>>>> Okay,
>>>>>>>
>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>
>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>> where you want them
>>>>>>>
>>>>>>> 3) Setup the solver as usual
>>>>>>>
>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>
>>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>>> are not used).
>>>>>> Please let me know what I should do now.
>>>>>>
>>>>>
>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>> partitioning cells, so you would have
>>>>> unknowns in cells. The code should look like this:
>>>>>
>>>>> PetscSectionCreate(comm, &s);
>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>> PetscSectionSetChart(s, 0, nC);
>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>       PetscInt point;
>>>>>
>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof are
>>>>> on each vertex
>>>>>     }
>>>>>   }
>>>>> }
>>>>> PetscSectionSetUp(s);
>>>>> DMSetDefaultSection(dm, s);
>>>>> PetscSectionDestroy(&s);
>>>>>
>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>
>>>>
>>>> I have put an example without the PetscSection here:
>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>> From the code you have written above, how do we let PetscSection select
>>>> only those cells that lie in the computational domain ?  Is it that for
>>>> every "point" inside the above loop, we check whether it lies in the domain
>>>> or not, e.g using the function isPosInDomain(...) in the .cxx file I linked
>>>> to?
>>>>
>>>
>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>
>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>
>>
>> Done.
>> I mask it out using isPosInDomain() function::
>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr);
>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>> You know how many dof are on each vertex
>>           }
>>
>> And please let me know when I can rebuild the 'next' branch of petsc so
>> that DMDAGetCellPoint can be used. Currently compiler complains as it
>> cannot find it.
>>
>
> Pushed.
>

Pulled it thanks, but compiler was complaining that it didn't find
DMDAGetCellPoint.
Doing grep -R 'DMDAGetCellPoint' include/
returned nothing although the implementation is there in dalocal.c
So I added the declaration to petscdmda.h file just above the declaration
of DMDAGetNumCells.

I have also added you as a collaborator in the github project repository so
that you can edit and add comments to the file I linked above. My
computeMatrix still uses the DMDA instead of the PetscSection, so is it
where the changes need to be made ?


>    Matt
>
>
>>
>>>    Matt
>>>
>>>
>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>>
>>>>>>>  If not, just put the identity into
>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>> scalability or convergence.
>>>>>>>>>
>>>>>>>>
>>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>
>>>>>>>
>>>>>>> Jed is talking about a constraint on a the pressure at a point. This
>>>>>>> is just decoupling these unknowns from the rest
>>>>>>> of the problem.
>>>>>>>
>>>>>>>
>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>> rid of the identity rows too.
>>>>>>>>
>>>>>>>
>>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>>> conditioning worse.
>>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>>> non-staggered case.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bishesh
>>>>>>>>
>>>>>>>>>
>>>>>>>>>   Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bishesh
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/ac85a9aa/attachment-0001.html>

From knepley at gmail.com  Mon Oct 28 15:00:17 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 28 Oct 2013 15:00:17 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
Message-ID: <CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>

On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>>
>>>
>>>
>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>>
>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <knepley at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear all,
>>>>>>>>>>> I would like to know if some of the petsc objects that I have
>>>>>>>>>>> not used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>>
>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>
>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>
>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>>>> the boundary points.
>>>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>>>> Later, I might have to extend this for the staggered grid case
>>>>>>>>>>> too.
>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can set
>>>>>>>>>>> up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>>
>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It sounds like the right solution for this is to use PetscSection
>>>>>>>>>> on top of DMDA. I am working on this, but it is really
>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>> development, we can help you.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>> great I think.
>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>>>> the development version before.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Okay,
>>>>>>>>
>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>
>>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>>> where you want them
>>>>>>>>
>>>>>>>> 3) Setup the solver as usual
>>>>>>>>
>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>
>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>>>> are not used).
>>>>>>> Please let me know what I should do now.
>>>>>>>
>>>>>>
>>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>>> partitioning cells, so you would have
>>>>>> unknowns in cells. The code should look like this:
>>>>>>
>>>>>> PetscSectionCreate(comm, &s);
>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>       PetscInt point;
>>>>>>
>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof are
>>>>>> on each vertex
>>>>>>     }
>>>>>>   }
>>>>>> }
>>>>>> PetscSectionSetUp(s);
>>>>>> DMSetDefaultSection(dm, s);
>>>>>> PetscSectionDestroy(&s);
>>>>>>
>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>
>>>>>
>>>>> I have put an example without the PetscSection here:
>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>> From the code you have written above, how do we let PetscSection
>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>> linked to?
>>>>>
>>>>
>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>
>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>
>>>
>>> Done.
>>> I mask it out using isPosInDomain() function::
>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr);
>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>> You know how many dof are on each vertex
>>>           }
>>>
>>> And please let me know when I can rebuild the 'next' branch of petsc so
>>> that DMDAGetCellPoint can be used. Currently compiler complains as it
>>> cannot find it.
>>>
>>
>> Pushed.
>>
>
> Pulled it thanks, but compiler was complaining that it didn't find
> DMDAGetCellPoint.
> Doing grep -R 'DMDAGetCellPoint' include/
> returned nothing although the implementation is there in dalocal.c
> So I added the declaration to petscdmda.h file just above the declaration
> of DMDAGetNumCells.
>

I will add the declaration.


> I have also added you as a collaborator in the github project repository
> so that you can edit and add comments to the file I linked above. My
> computeMatrix still uses the DMDA instead of the PetscSection, so is it
> where the changes need to be made ?
>

Instead of MatSetValuesStencil(), you would go back to using
MatSetValues(), and calculate the indices using PetscSection.
You get the section

  DMGetDefaultGlobalSection(dm, &gs)

and then

  DMDAGetCellPoint(dm, i, j, k, &p);
  PetscSectionGetOffset(gs, p, &ind);
  row = ind < 0 ? -(ind+1) : ind;

   Thanks,

     Matt


>
>>    Matt
>>
>>
>>>
>>>>    Matt
>>>>
>>>>
>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>>
>>>>>>>>  If not, just put the identity into
>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>> scalability or convergence.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>
>>>>>>>>
>>>>>>>> Jed is talking about a constraint on a the pressure at a point.
>>>>>>>> This is just decoupling these unknowns from the rest
>>>>>>>> of the problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>> rid of the identity rows too.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>>>> conditioning worse.
>>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>> non-staggered case.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bishesh
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>   Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Bishesh
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/1e1d0584/attachment.html>

From lu_qin_2000 at yahoo.com  Mon Oct 28 16:45:30 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Mon, 28 Oct 2013 14:45:30 -0700 (PDT)
Subject: [petsc-users] Could not find make
	when	building	PETSc	in	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
Message-ID: <1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>

Satish,

I followed your suggestion to run "export PATH=/usr/bin:$PATH", now grep, rm, etc., point to those under /usr/bin. I then run configure and make all again, the previous make errors about grep, etc. , were gone! Now the make all ran to the end but gave the error "makefile:24: recipe for target 'all' failed", as shown below.? I opened the error.log and it said

?"/usr/bin/ar: petscvecmod.o: No such file or directory
/usr/bin/cp: cannot stat `*.mod': No such file or directory"

But I checked "which ar" and it returned "/usr/bin/ar", so ar.exe is there.

The error.log and make.log?makefile are attached for details.

Thanks for any suggestions,
Qin?

=======? make?output ========
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d/comp
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d/incomp
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/benchmarks
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude/ftn-auto
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude/ftn-custom
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/petsc-private
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/sieve
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/tutorials
libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/tutorials/python
Completed building libraries
=========================================
**************************ERROR************************************
Error during compile, check arch-mswin-c-opt/conf/make.log
Send it and arch-mswin-c-opt/conf/configure.log to petsc-maint at mcs.anl.gov
*****************************************************************
makefile:24: recipe for target 'all' failedmake: *** [all] Error 1



On Monday, October 28, 2013 11:50 AM, Satish Balay <balay at mcs.anl.gov> wrote:

cygwin should have already setup PATH correctly so that its listed first

What do you have for:

echo $PATH
which make
which rm
which grep
export PATH=/usr/bin:$PATH
which make
which rm
which grep

Satish



On Mon, 28 Oct 2013, Qin Lu wrote:

> Satish,
> ?
> As far as I know, cygwin reads the Windows enviromental variable PATH. Can I not change the the setting in in the Windows enviromental variables (since other programs need them), but change the PATH used by cygwin only? Which file should I change?
> ?
> Thanks,
> Qin
> 
> 
> 
> On Monday, October 28, 2013 11:16 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>? 
> > /cygdrive/c/Windows/SUA/common/rm -f -rf /cygdrive/c/cygwin_cache/petsc-3.4.3/arch-mswin-c-opt/lib/libpetsc*.*
> 
> You appear to have non cygwin stuff in your PATH. Can you remove these
> things from your PATH - make sure grep in PATH is from cygwin - and
> rerun configure && make
> 
> Satish
> 
> 
> 
> On Mon, 28 Oct 2013, Qin Lu wrote:
> 
> > Hi,
> > ?
> > The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
> > ?
> > I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
> > ?
> > Thanks for?any hints,
> > Qin?
> > ?
> > ?
> > 
> > 
> > 
> > On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >? 
> > pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.
> > 
> > It will go into 'maint' later..
> > 
> > Satish
> > 
> > 
> > On Thu, 24 Oct 2013, Satish Balay wrote:
> > 
> > > Perhaps we should add back the windows-python check?
> > > 
> > > [since we haven't yet added the windows python support]
> > > 
> > > Satish
> > > 
> > > On Thu, 24 Oct 2013, Qin Lu wrote:
> > > 
> > > > Chetan
> > > > 
> > > > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > > > 
> > > > Thanks a lot for all you guys'?helps.
> > > > 
> > > > Qin?
> > > > 
> > > > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > > > I vaguely recall seeing the "could not locate make" error
> > > > when a non-cygwin python is picked up first in path.
> > > > So ensure that "which python" points to cygwin python.
> > > > 
> > > > Another issue is that the --with-make argument, if it is
> > > > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > > > other drive letter in between where cygwin is installed).
> > > > 
> > > > Chetan
> > > > 
> > > > 
> > > > > -----Original Message-----
> > > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > > > Sent: Thursday, October 24, 2013 8:43 AM
> > > > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > > > 
> > > > > Jed,
> > > > > 
> > > > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > > > in confugure, it gave the same error.
> > > > > 
> > > > > Thanks,
> > > > > Qin
> > > > > 
> > > > > 
> > > > > 
> > > > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > > > 
> > > > > > Hello,
> > > > > >
> > > > > > I am trying to build PETSc in Windows 7 with following steps:
> > > > > >
> > > > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > > > >
> > > > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > > > 
> > > > > You need to pass the full path to the make executable (or add it to your
> > > > > path), or you can use --download-make.
> > > > > 
> > > > > 
> > > > > > What should I do to let configure know the path of make?
> > > > > >
> > > > > > Many thanks for your help,
> > > > > >
> > > > > > Regard,
> > > > > > Qin
> > > > 
> > >????????
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error.log
Type: application/octet-stream
Size: 7996 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/ef214019/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 124353 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131028/ef214019/attachment-0003.obj>

From balay at mcs.anl.gov  Mon Oct 28 19:34:46 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 28 Oct 2013 19:34:46 -0500 (CDT)
Subject: [petsc-users] Could not find make when building PETSc in
 Windows 7
In-Reply-To: <1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
	<1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310281926210.3576@mcs.anl.gov>

You appear to have 2 issues:


>>>>>>>>
fp.c^M
C:\CYGWIN~1\PETSC-~1.2\src\sys\error\fp.c(425): error: expression must have arithmetic or pointer type^M
      if (feclearexcept(FE_ALL_EXCEPT)) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"Cannot clear floating point exception flags\n");^M
          ^^M
^M
C:\CYGWIN~1\PETSC-~1.2\src\sys\error\fp.c(436): error: expression must have arithmetic or pointer type^M
      if (fesetenv(FE_DFL_ENV)) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_LIB,"Cannot disable floating point exceptions");^M

<<<<<<<<<<

Looks like icc on windows has fenv.h - but doesn't support exceptions
properly. To workarround this - edit 
/cygdrive/c/cygwin_cache/petsc-3.4.2/arch-mswin-c-opt/include
and remove  the lines related to PETSC_HAVE_FENV_H

>>>>>
  "/usr/bin/ar: petscvecmod.o: No such file or directory
<<<<<

Mostlikely this is related to using 'winzip' or '7zip' to untar petsc
sources. Try using 'tar' from cygwin.

Or you can ignore this issue if not using f90 modules from your code.

Satish

On Mon, 28 Oct 2013, Qin Lu wrote:

> Satish,
> 
> I followed your suggestion to run "export PATH=/usr/bin:$PATH", now grep, rm, etc., point to those under /usr/bin. I then run configure and make all again, the previous make errors about grep, etc. , were gone! Now the make all ran to the end but gave the error "makefile:24: recipe for target 'all' failed", as shown below.? I opened the error.log and it said
> 
> ?"/usr/bin/ar: petscvecmod.o: No such file or directory
> /usr/bin/cp: cannot stat `*.mod': No such file or directory"
> 
> But I checked "which ar" and it returned "/usr/bin/ar", so ar.exe is there.
> 
> The error.log and make.log?makefile are attached for details.
> 
> Thanks for any suggestions,
> Qin?
> 
> =======? make?output ========
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d/comp
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/contrib/fun3d/incomp
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/src/benchmarks
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude/ftn-auto
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/finclude/ftn-custom
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/petsc-private
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/include/sieve
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/tutorials
> libfast in: /cygdrive/c/cygwin_cache/petsc-3.4.2/tutorials/python
> Completed building libraries
> =========================================
> **************************ERROR************************************
> Error during compile, check arch-mswin-c-opt/conf/make.log
> Send it and arch-mswin-c-opt/conf/configure.log to petsc-maint at mcs.anl.gov
> *****************************************************************
> makefile:24: recipe for target 'all' failedmake: *** [all] Error 1
> 
> 
> 
> On Monday, October 28, 2013 11:50 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> cygwin should have already setup PATH correctly so that its listed first
> 
> What do you have for:
> 
> echo $PATH
> which make
> which rm
> which grep
> export PATH=/usr/bin:$PATH
> which make
> which rm
> which grep
> 
> Satish
> 
> 
> 
> On Mon, 28 Oct 2013, Qin Lu wrote:
> 
> > Satish,
> > ?
> > As far as I know, cygwin reads the Windows enviromental variable PATH. Can I not change the the setting in in the Windows enviromental variables (since other programs need them), but change the PATH used by cygwin only? Which file should I change?
> > ?
> > Thanks,
> > Qin
> > 
> > 
> > 
> > On Monday, October 28, 2013 11:16 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> >? 
> > > /cygdrive/c/Windows/SUA/common/rm -f -rf /cygdrive/c/cygwin_cache/petsc-3.4.3/arch-mswin-c-opt/lib/libpetsc*.*
> > 
> > You appear to have non cygwin stuff in your PATH. Can you remove these
> > things from your PATH - make sure grep in PATH is from cygwin - and
> > rerun configure && make
> > 
> > Satish
> > 
> > 
> > 
> > On Mon, 28 Oct 2013, Qin Lu wrote:
> > 
> > > Hi,
> > > ?
> > > The configuration seemed to work, then I ran "make all" but got errors such as "grep: Cannot open file '/dev/fs/C/cygdrive/c/cygwin_cache/petsc-3.4.3/include/petscversion.h': No such file or directory". See the attached error.log and make.log for details.
> > > ?
> > > I confirmed that c:\cygwin_cache\petsc-3.4.3\include\petscversion.h is there. '\The ''/dev/fs/C' part in the directory seems?suspicious. Did I missed anything in the setup?
> > > ?
> > > Thanks for?any hints,
> > > Qin?
> > > ?
> > > ?
> > > 
> > > 
> > > 
> > > On Thursday, October 24, 2013 2:05 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > >? 
> > > pushed ee76e990026b11c1f9c153f288b72cc9b8ebb100 and merged to next.
> > > 
> > > It will go into 'maint' later..
> > > 
> > > Satish
> > > 
> > > 
> > > On Thu, 24 Oct 2013, Satish Balay wrote:
> > > 
> > > > Perhaps we should add back the windows-python check?
> > > > 
> > > > [since we haven't yet added the windows python support]
> > > > 
> > > > Satish
> > > > 
> > > > On Thu, 24 Oct 2013, Qin Lu wrote:
> > > > 
> > > > > Chetan
> > > > > 
> > > > > Great, that?was the problem! I set python point?to?a?copy that was not?cygwin's. Now I?set it to cygwin's python and it this error is gone! I am waiting to see if the whole building will finish without problems.
> > > > > 
> > > > > Thanks a lot for all you guys'?helps.
> > > > > 
> > > > > Qin?
> > > > > 
> > > > > On Thursday, October 24, 2013 11:28 AM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> > > > > I vaguely recall seeing the "could not locate make" error
> > > > > when a non-cygwin python is picked up first in path.
> > > > > So ensure that "which python" points to cygwin python.
> > > > > 
> > > > > Another issue is that the --with-make argument, if it is
> > > > > needed, should be /cygdrive/C/cygwin64/bin (or with some
> > > > > other drive letter in between where cygwin is installed).
> > > > > 
> > > > > Chetan
> > > > > 
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Qin Lu
> > > > > > Sent: Thursday, October 24, 2013 8:43 AM
> > > > > > To: Jed Brown; petsc-users at mcs.anl.gov
> > > > > > Subject: Re: [petsc-users] Could not find make when building PETSc in Windows 7
> > > > > > 
> > > > > > Jed,
> > > > > > 
> > > > > > As I mentioned, I did add C:\cygwin64\bin (where make.exe is) to Path, and also tried to run
> > > > > > ./configure --with-cc='win32fe icl' --with-fc='win32fe ifort' --with-cxx='win32fe icl' --with-
> > > > > > make=/cygdrive/cygwin64/bin. It did not work, I just tried what you suggested: adding --download-make
> > > > > > in confugure, it gave the same error.
> > > > > > 
> > > > > > Thanks,
> > > > > > Qin
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > On Thursday, October 24, 2013 10:36 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > > > > > Qin Lu <lu_qin_2000 at yahoo.com> writes:
> > > > > > 
> > > > > > > Hello,
> > > > > > >
> > > > > > > I am trying to build PETSc in Windows 7 with following steps:
> > > > > > >
> > > > > > > 1. Installed cygwin 64-bit with python, make and other default packages under C:\cygwin64
> > > > > > > 2. Added C:\cygwin64\bin to Windows environmental variable Path.
> > > > > > > 3. Downloaded and unzipped PETSc 3.4.2.
> > > > > > > 4. Run mintty.exe to open a cygwin terminal.
> > > > > > > 5. In the cygwin terminal, under?PETSc directory,?run:??./configure --with-cc='win32fe icl' --with-
> > > > > > fc='win32fe ifort' --with-cxx='win32fe icl'
> > > > > > >
> > > > > > > However, it gave an error "Could not locate the make utility on your system, make sure it is in your
> > > > > > path or use --with-make=/fullpathnameofmake and run ./configure again".??I suppose it should use
> > > > > > C:\cygwin64\bin\make.exe, which is already in path, but?I?still added? --with-
> > > > > > make=/cvgdrive/cygwin64/bin in?configure's option and run configure?again, yet?it gave the same error.
> > > > > > 
> > > > > > You need to pass the full path to the make executable (or add it to your
> > > > > > path), or you can use --download-make.
> > > > > > 
> > > > > > 
> > > > > > > What should I do to let configure know the path of make?
> > > > > > >
> > > > > > > Many thanks for your help,
> > > > > > >
> > > > > > > Regard,
> > > > > > > Qin
> > > > > 
> > > >????????

From bisheshkh at gmail.com  Tue Oct 29 05:31:37 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Tue, 29 Oct 2013 11:31:37 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
Message-ID: <CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>

On Mon, Oct 28, 2013 at 9:00 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>>
>>
>>
>> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Dear all,
>>>>>>>>>>>> I would like to know if some of the petsc objects that I have
>>>>>>>>>>>> not used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>>>
>>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>>
>>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>>
>>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>>> But in this matrix it should have the rows corresponding to the
>>>>>>>>>>>> nodes only on the computational domain. It would be nice if I can easily
>>>>>>>>>>>> (using (i,j,k) indexing) put on the rhs dirichlet values corresponding to
>>>>>>>>>>>> the boundary points.
>>>>>>>>>>>> Then, once the system is solved, put the values of the solution
>>>>>>>>>>>> back to the corresponding positions in the binary image.
>>>>>>>>>>>> Later, I might have to extend this for the staggered grid case
>>>>>>>>>>>> too.
>>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can
>>>>>>>>>>>> set up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>>>
>>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It sounds like the right solution for this is to use
>>>>>>>>>>> PetscSection on top of DMDA. I am working on this, but it is really
>>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>>> development, we can help you.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>>> great I think.
>>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not used
>>>>>>>>>> the development version before.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Okay,
>>>>>>>>>
>>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>>
>>>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>>>> where you want them
>>>>>>>>>
>>>>>>>>> 3) Setup the solver as usual
>>>>>>>>>
>>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>>
>>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>>>>> are not used).
>>>>>>>> Please let me know what I should do now.
>>>>>>>>
>>>>>>>
>>>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>>>> partitioning cells, so you would have
>>>>>>> unknowns in cells. The code should look like this:
>>>>>>>
>>>>>>> PetscSectionCreate(comm, &s);
>>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>>       PetscInt point;
>>>>>>>
>>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof
>>>>>>> are on each vertex
>>>>>>>     }
>>>>>>>   }
>>>>>>> }
>>>>>>> PetscSectionSetUp(s);
>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>> PetscSectionDestroy(&s);
>>>>>>>
>>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>>
>>>>>>
>>>>>> I have put an example without the PetscSection here:
>>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>>> From the code you have written above, how do we let PetscSection
>>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>>> linked to?
>>>>>>
>>>>>
>>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>>
>>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>>
>>>>
>>>> Done.
>>>> I mask it out using isPosInDomain() function::
>>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr);
>>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>>> You know how many dof are on each vertex
>>>>           }
>>>>
>>>> And please let me know when I can rebuild the 'next' branch of petsc so
>>>> that DMDAGetCellPoint can be used. Currently compiler complains as it
>>>> cannot find it.
>>>>
>>>
>>> Pushed.
>>>
>>
>> Pulled it thanks, but compiler was complaining that it didn't find
>> DMDAGetCellPoint.
>> Doing grep -R 'DMDAGetCellPoint' include/
>> returned nothing although the implementation is there in dalocal.c
>> So I added the declaration to petscdmda.h file just above the declaration
>> of DMDAGetNumCells.
>>
>
> I will add the declaration.
>
>
>> I have also added you as a collaborator in the github project repository
>> so that you can edit and add comments to the file I linked above. My
>> computeMatrix still uses the DMDA instead of the PetscSection, so is it
>> where the changes need to be made ?
>>
>
> Instead of MatSetValuesStencil(), you would go back to using
> MatSetValues(), and calculate the indices using PetscSection.
> You get the section
>
>   DMGetDefaultGlobalSection(dm, &gs)
>
> and then
>
>   DMDAGetCellPoint(dm, i, j, k, &p);
>   PetscSectionGetOffset(gs, p, &ind);
>   row = ind < 0 ? -(ind+1) : ind;
>
>

I'm sorry but did not understood. For e.g., in 2D, for each point p, I have
2 dof. Does PetsSectionGetOffset give me the two offsets for the dof sth
like ind[0] = 0 and ind[1] = 1 ?
How would you change the following code that sets 2D laplacian stencil  for
2 dof to use petscsection and Matsetvalues instead ?

 ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
    ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
    ierr = DMGetDefaultGlobalSection(dm,&gs);CHKERRQ(ierr); /*Here I get
the PetscSection*/
    for(PetscInt dof = 0; dof < 2; ++dof) {
        row.c = dof;
        for(PetscInt k = 0; k < 5; ++k) {
            col[k].c = dof;
        }
        for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
            for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
                num = 0;
                /*ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/
/*Get the PetscPoint*/
                /*But I did now understand how I would emulate the row.c
and col.c info with PetscSection*/
                row.i = i;  row.j = j;
                col[num].i = i; col[num].j = j;
                if (isPosInDomain(ctx,i,j)) {      /*put the operator for
only the values for only the points lying in the domain*/
                    v[num++] = -4;
                    if(isPosInDomain(ctx,i+1,j)) {
                        col[num].i = i+1;   col[num].j = j;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i-1,j)) {
                        col[num].i = i-1;   col[num].j = j;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i,j+1)) {
                        col[num].i = i;     col[num].j = j+1;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i,j-1)) {
                        col[num].i = i;     col[num].j = j-1;
                        v[num++] = 1;
                    }
                } else {
                    v[num++] = dScale;  /*set Dirichlet identity rows for
points in the rectangle but outside the computational domain*/
                }
                ierr =
MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
            }
        }
    }


>    Thanks,
>
>      Matt
>
>
>>
>>>    Matt
>>>
>>>
>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>>
>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In the case of Poisson with Dirichlet condition this might be the
>>>>>>>>>> case. But is it always true that having identity rows in the system matrix
>>>>>>>>>> will not hurt convergence ? I thought otherwise for the following reasons:
>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jed is talking about a constraint on a the pressure at a point.
>>>>>>>>> This is just decoupling these unknowns from the rest
>>>>>>>>> of the problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It can change the exact iteration, but it does not make the matrix
>>>>>>>>> conditioning worse.
>>>>>>>>>
>>>>>>>>>    Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  Anyway please provide me with some pointers so that I can start
>>>>>>>>>> trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>> non-staggered case.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bishesh
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>   Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/44c74ce8/attachment-0001.html>

From christophe.ortiz at ciemat.es  Tue Oct 29 05:43:14 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Tue, 29 Oct 2013 11:43:14 +0100
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <87ppqp1omf.fsf@mcs.anl.gov>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
	<CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
	<87txg53ymx.fsf@mcs.anl.gov>
	<CANBrw+5Rb5_MaEauJdz7ETczfUu-cf3zoB6nbuAs8FNvu3ZGwg@mail.gmail.com>
	<87ppqp1omf.fsf@mcs.anl.gov>
Message-ID: <CANBrw+4ocUXH6_7dRmP1uzxWsTKoowf0iR3hjs+g_HSFQU+cPA@mail.gmail.com>

On Mon, Oct 28, 2013 at 3:00 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> > But just to understand what I'm doing, what does TSSetEquationType do
> > internally ?
>
> It tells the method what is necessary to evaluate stage derivatives, and
> affects the starting method and (possibly) what can be carried over from
> the last step.
>
>
Ok, I understand. For the moment it works well with this option.

I'm still doing some tests and I would like to know if it's possible to
view a solution in log scale ?
To view the solution after diffusion I use
 ierr = PetscViewerDrawOpen(PETSC_COMM_WORLD,NULL,"",0,0,300,300,&viewer);
VecView(X,viewer);

Since the values of X are large (~1e20) and vary logarithmically, I would
like to view them in log scale. How can I do that ?

Also, is it possible to view the "real coordinates" in x-axis, instead of
the node number ?

Christophe



> > For the type of problem I'm trying to solve, is TS_EQ_DAE_IMPLICIT_INDEX1
> > ok ?
>
> Yes.
>
> > What is the difference with TS_EQ_DAE_IMPLICIT_INDEX2,3 or _INDEXHI ?
>
> I believe these are currently unused, but if we add support for
> computing consistent initial conditions, we'll have to know the index.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/df296c5c/attachment.html>

From allenbearing05 at vip.163.com  Tue Oct 29 08:43:43 2013
From: allenbearing05 at vip.163.com (allenbearing)
Date: Tue, 29 Oct 2013 21:43:43 +0800 (CST)
Subject: [petsc-users] Rolamento
Message-ID: <237d5c29.1594c.1420474a9ee.Coremail.allenbearing05@vip.163.com>

Prezados Senhores e Senhora:


Sauda??es.

Fico feliz em saber que voc? est? no mercado para mancais. como voc? pode saber que se especializam na exporta??o e fornecimento de NSK, FAG, INA, SKF, IKO, NTN, KOYO, Nachi, TIMKEN e assim por diante mais de 30 anos em Xangai, China., com a for?a de um bom pre?o e entrega r?pida (mais de modelos pode ser a entrega no prazo de 1 semana). estamos se NO.1 fornecedor rolamento / exportador de Xangai agora.

Se voc? tiver alguma d?vida, n?o hesite em enviar-me retornar e-mail ou voc? pode navegar em nosso site:. Www.allen-bearing.com para mais informa??es sobre o produto.

cumprimentos

Allen Wang
Gerente de venda
Xangai Allen Bearing Co., Ltd
Adicionar: Baolv Road.Baoshan Disctrict.P.R.Of China
T. 0086-510-82417855 0086-189-1719-1917
Email: allenbearing at allen-bearing.com
SKYPE: hymotorcycle
Web: www.allen-bearing.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/cbff3e59/attachment.html>

From cedric.doucet at inria.fr  Tue Oct 29 10:24:07 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Tue, 29 Oct 2013 16:24:07 +0100 (CET)
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <CAMYG4G=GdmQAHBJAiDCM=_A_4y9zYtEg_ou-zrY6M=oHPmFebg@mail.gmail.com>
Message-ID: <60378487.100462.1383060247801.JavaMail.root@inria.fr>

Hello, 

I have a short question about the code I need to modify. 
As far as I understand, numCorner should be replaced by an array numCorners[numCells] containing the number of vertices of each cell. 
The body of DMPlexBuildFromCellList_Private function becomes 
--------------------------------------------------------------------------------------------------------------------------------------- 
ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr); 
for (c = 0; c < numCells; ++c) { 
ierr = DMPlexSetConeSize(dm, c, numCorners[c]);CHKERRQ(ierr); 
} 
ierr = DMSetUp(dm);CHKERRQ(ierr); 

for (c = 0; c < numCells; ++c) { 
ierr = DMGetWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr); 
for (p = 0; p < numCorners[c]; ++p) { 
cone[p] = cells[c*numCorners[c]+p]+numCells; 
} 
ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr); 

ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr); 
} 

ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr); 
ierr = DMPlexStratify(dm);CHKERRQ(ierr); 
------------------------------------------------------------------------------------------------------------------------------------------ 
However, I am not quite sure that DMGetWorkArray and DMRestoreWorkArray should be used like this. 
I guess that DMRestoreWorkArray behaves like free function in ansi c but I am not sure. 
Do you think that calling DMGetWorkArray and DMRestoreWorkArray inside a loop is a good thing from the point of view of computational time and memory management? 

Note: the best way to do this may be to first sort cells by numCorners to avoid many calls to DMGetWorkArray and DMRestoreWorkArray. This is actually what I have in my own code but I would like to maintain Petsc's philosophy. 

Thanks, 

C?dric 

----- Mail original -----

> De: "Matthew Knepley" <knepley at gmail.com>
> ?: "Cedric Doucet" <cedric.doucet at inria.fr>
> Cc: petsc-users at mcs.anl.gov
> Envoy?: Vendredi 25 Octobre 2013 16:31:59
> Objet: Re: [petsc-users] DMPlexCreateFromCellList and hybrid meshes

> On Fri, Oct 25, 2013 at 7:23 AM, Cedric Doucet <
> cedric.doucet at inria.fr > wrote:

> > Hello,
> 

> > I've noticed that DMPlexCreateFromCellList assumes that cells have
> > the same number of vertices (numcorners argument).
> 
> > What should be done when one wants to create a DMPlex from a mesh
> > containing different types of cells?
> 
> > Does one have to create several DMPlex structures and merge them?
> 
> > Does one have to create a unique DMPlex by hand?
> 

> The code is very short:

> ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
> for (c = 0; c < numCells; ++c) {
> ierr = DMPlexSetConeSize(dm, c, numCorners);CHKERRQ(ierr);
> }
> ierr = DMSetUp(dm);CHKERRQ(ierr);
> ierr = DMGetWorkArray(dm, numCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
> for (c = 0; c < numCells; ++c) {
> for (p = 0; p < numCorners; ++p) {
> cone[p] = cells[c*numCorners+p]+numCells;
> }
> ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
> }
> ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
> ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
> ierr = DMPlexStratify(dm);CHKERRQ(ierr);

> This code is all in plexcreate.c. If you want different cells, you
> can change numCorners for each cell.
> I could make a convenience form if necessary.

> Matt

> > Thank you very much for your help.
> 

> > C?dric
> 

> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/6fc897dc/attachment.html>

From knepley at gmail.com  Tue Oct 29 11:24:03 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Oct 2013 11:24:03 -0500
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <60378487.100462.1383060247801.JavaMail.root@inria.fr>
References: <CAMYG4G=GdmQAHBJAiDCM=_A_4y9zYtEg_ou-zrY6M=oHPmFebg@mail.gmail.com>
	<60378487.100462.1383060247801.JavaMail.root@inria.fr>
Message-ID: <CAMYG4G=z+FWqPkP8+eELUicvCu3eEdCF_FcZ_6bRTb7Er+2rrA@mail.gmail.com>

On Tue, Oct 29, 2013 at 10:24 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:

> Hello,
>
> I have a short question about the code I need to modify.
> As far as I understand, numCorner should be replaced by an array
> numCorners[numCells] containing the number of vertices of each cell.
> The body of DMPlexBuildFromCellList_Private function becomes
>
> ---------------------------------------------------------------------------------------------------------------------------------------
> ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
>
    maxCorners = 0;

>   for (c = 0; c < numCells; ++c) {
>     ierr = DMPlexSetConeSize(dm, c, numCorners[c]);CHKERRQ(ierr);
>
         maxCorners = PetscMax(maxCorners, numCorners[c]);

>   }
>   ierr = DMSetUp(dm);CHKERRQ(ierr);
>

    ierr = DMGetWorkArray(dm, maxCorners, PETSC_INT, &cone);CHKERRQ(ierr);

>   for (c = 0, off = 0; c < numCells; ++c) {
>     for (p = 0; p < numCorners[c]; ++p) {
>       cone[p] = cells[off+p]+numCells;
>     }
>     ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
>
         off += numCorners[c];

>   }
>
    ierr = DMRestoreWorkArray(dm, maxCorners, PETSC_INT,
&cone);CHKERRQ(ierr);

>

>
>   ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
>   ierr = DMPlexStratify(dm);CHKERRQ(ierr);
>

   Matt


>
> ------------------------------------------------------------------------------------------------------------------------------------------
> However, I am not quite sure that DMGetWorkArray and DMRestoreWorkArray
> should be used like this.
> I guess that DMRestoreWorkArray behaves like free function in ansi c but I
> am not sure.
> Do you think that calling DMGetWorkArray and DMRestoreWorkArray inside a
> loop is a good thing from the point of view of computational time and
> memory management?
>
> Note: the best way to do this may be to first sort cells by numCorners to
> avoid many calls to DMGetWorkArray and DMRestoreWorkArray. This is actually
> what I have in my own code but I would like to maintain Petsc's philosophy.
>
> Thanks,
>
> C?dric
>
>
>
>
>
> ------------------------------
>
> *De: *"Matthew Knepley" <knepley at gmail.com>
> *?: *"Cedric Doucet" <cedric.doucet at inria.fr>
> *Cc: *petsc-users at mcs.anl.gov
> *Envoy?: *Vendredi 25 Octobre 2013 16:31:59
> *Objet: *Re: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
>
> On Fri, Oct 25, 2013 at 7:23 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:
>
>> Hello,
>>
>> I've noticed that DMPlexCreateFromCellList assumes that cells have the
>> same number of vertices (numcorners argument).
>> What should be done when one wants to create a DMPlex from a mesh
>> containing different types of cells?
>> Does one have to create several DMPlex structures and merge them?
>> Does one have to create a unique DMPlex by hand?
>>
>
> The code is very short:
>
>   ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
>   for (c = 0; c < numCells; ++c) {
>     ierr = DMPlexSetConeSize(dm, c, numCorners);CHKERRQ(ierr);
>   }
>   ierr = DMSetUp(dm);CHKERRQ(ierr);
>   ierr = DMGetWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr);
>   for (c = 0; c < numCells; ++c) {
>     for (p = 0; p < numCorners; ++p) {
>       cone[p] = cells[c*numCorners+p]+numCells;
>     }
>     ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
>   }
>   ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
>   ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
>   ierr = DMPlexStratify(dm);CHKERRQ(ierr);
>
> This code is all in plexcreate.c. If you want different cells, you can
> change numCorners for each cell.
> I could make a convenience form if necessary.
>
>    Matt
>
>
>>
>> Thank you very much for your help.
>>
>> C?dric
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/87d18492/attachment-0001.html>

From knepley at gmail.com  Tue Oct 29 11:37:55 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 Oct 2013 11:37:55 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
Message-ID: <CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>

On Tue, Oct 29, 2013 at 5:31 AM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> On Mon, Oct 28, 2013 at 9:00 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>>
>>>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <
>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>> I would like to know if some of the petsc objects that I have
>>>>>>>>>>>>> not used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>>>> The binary information of the image partitions the cube into a
>>>>>>>>>>>>> computational domain and non-computational domain.
>>>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>>>> But in this matrix it should have the rows corresponding to
>>>>>>>>>>>>> the nodes only on the computational domain. It would be nice if I can
>>>>>>>>>>>>> easily (using (i,j,k) indexing) put on the rhs dirichlet values
>>>>>>>>>>>>> corresponding to the boundary points.
>>>>>>>>>>>>> Then, once the system is solved, put the values of the
>>>>>>>>>>>>> solution back to the corresponding positions in the binary image.
>>>>>>>>>>>>> Later, I might have to extend this for the staggered grid case
>>>>>>>>>>>>> too.
>>>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can
>>>>>>>>>>>>> set up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It sounds like the right solution for this is to use
>>>>>>>>>>>> PetscSection on top of DMDA. I am working on this, but it is really
>>>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>>>> development, we can help you.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>>>> great I think.
>>>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not
>>>>>>>>>>> used the development version before.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Okay,
>>>>>>>>>>
>>>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>>>
>>>>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>>>>> where you want them
>>>>>>>>>>
>>>>>>>>>> 3) Setup the solver as usual
>>>>>>>>>>
>>>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>>>
>>>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>>>>>> are not used).
>>>>>>>>> Please let me know what I should do now.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>>>>> partitioning cells, so you would have
>>>>>>>> unknowns in cells. The code should look like this:
>>>>>>>>
>>>>>>>> PetscSectionCreate(comm, &s);
>>>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>>>       PetscInt point;
>>>>>>>>
>>>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof
>>>>>>>> are on each vertex
>>>>>>>>     }
>>>>>>>>   }
>>>>>>>> }
>>>>>>>> PetscSectionSetUp(s);
>>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>>> PetscSectionDestroy(&s);
>>>>>>>>
>>>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>>>
>>>>>>>
>>>>>>> I have put an example without the PetscSection here:
>>>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>>>> From the code you have written above, how do we let PetscSection
>>>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>>>> linked to?
>>>>>>>
>>>>>>
>>>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>>>
>>>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>>>
>>>>>
>>>>> Done.
>>>>> I mask it out using isPosInDomain() function::
>>>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr
>>>>> );
>>>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>>>> You know how many dof are on each vertex
>>>>>           }
>>>>>
>>>>> And please let me know when I can rebuild the 'next' branch of petsc
>>>>> so that DMDAGetCellPoint can be used. Currently compiler complains as it
>>>>> cannot find it.
>>>>>
>>>>
>>>> Pushed.
>>>>
>>>
>>> Pulled it thanks, but compiler was complaining that it didn't find
>>> DMDAGetCellPoint.
>>> Doing grep -R 'DMDAGetCellPoint' include/
>>> returned nothing although the implementation is there in dalocal.c
>>> So I added the declaration to petscdmda.h file just above the
>>> declaration of DMDAGetNumCells.
>>>
>>
>> I will add the declaration.
>>
>>
>>>  I have also added you as a collaborator in the github project
>>> repository so that you can edit and add comments to the file I linked
>>> above. My computeMatrix still uses the DMDA instead of the PetscSection, so
>>> is it where the changes need to be made ?
>>>
>>
>> Instead of MatSetValuesStencil(), you would go back to using
>> MatSetValues(), and calculate the indices using PetscSection.
>> You get the section
>>
>>   DMGetDefaultGlobalSection(dm, &gs)
>>
>> and then
>>
>>   DMDAGetCellPoint(dm, i, j, k, &p);
>>   PetscSectionGetOffset(gs, p, &ind);
>>   row = ind < 0 ? -(ind+1) : ind;
>>
>>
>
> I'm sorry but did not understood. For e.g., in 2D, for each point p, I
> have 2 dof. Does PetsSectionGetOffset give me the two offsets for the dof
> sth like ind[0] = 0 and ind[1] = 1 ?
> How would you change the following code that sets 2D laplacian stencil
> for 2 dof to use petscsection and Matsetvalues instead ?
>

Forget about multiple degrees of freedom until your scalar problem works.


>  ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>     ierr = DMGetDefaultGlobalSection(dm,&gs);CHKERRQ(ierr); /*Here I get
> the PetscSection*/
>     for(PetscInt dof = 0; dof < 2; ++dof) {
>         row.c = dof;
>         for(PetscInt k = 0; k < 5; ++k) {
>             col[k].c = dof;
>         }
>

I can't imagine that this is what you want. You have two non-interacting
problems.


>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>                 num = 0;
>                 /*ierr =
> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>                 /*But I did now understand how I would emulate the row.c
> and col.c info with PetscSection*/
>                 row.i = i;  row.j = j;
>

Here you would call

   ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
   ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
   ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);

This means that this cell has global rows = [row, row+dof).


>                 col[num].i = i; col[num].j = j;
>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator for
> only the values for only the points lying in the domain*/
>                     v[num++] = -4;
>                     if(isPosInDomain(ctx,i+1,j)) {
>                         col[num].i = i+1;   col[num].j = j;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i-1,j)) {
>                         col[num].i = i-1;   col[num].j = j;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i,j+1)) {
>                         col[num].i = i;     col[num].j = j+1;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i,j-1)) {
>                         col[num].i = i;     col[num].j = j-1;
>                         v[num++] = 1;
>                     }
>                 } else {
>                     v[num++] = dScale;  /*set Dirichlet identity rows for
> points in the rectangle but outside the computational domain*/
>                 }
>

You do the same thing for the columns, and then call

  ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
INSERT_VALUES);CHKERRQ(ierr);


>                 ierr =
> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>             }
>         }
>     }
>
>
>>    Thanks,
>>
>>      Matt
>>
>>
>>>
>>>>    Matt
>>>>
>>>>
>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might be
>>>>>>>>>>> the case. But is it always true that having identity rows in the system
>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>> reasons:
>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jed is talking about a constraint on a the pressure at a point.
>>>>>>>>>> This is just decoupling these unknowns from the rest
>>>>>>>>>> of the problem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>
>>>>>>>>>>    Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Bishesh
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>   Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/817326df/attachment-0001.html>

From danyang.su at gmail.com  Tue Oct 29 18:02:51 2013
From: danyang.su at gmail.com (Danyang Su)
Date: Tue, 29 Oct 2013 16:02:51 -0700
Subject: [petsc-users] Unresolved external symbol in PETSc for Windows
In-Reply-To: <000001ced4a9$f2e8fd00$d8baf700$@mic-tc.ch>
References: <000001ced4a9$f2e8fd00$d8baf700$@mic-tc.ch>
Message-ID: <52703E9B.7040103@gmail.com>

Hi All,

I am currently testing a PETSc fortran project for Windows. The package 
I use is "PETSc for Windows 2.0" (see bellow) which can be used on 
windows without installing CYGWIN on windows. I build a C project using 
the example ex2.c and it works fine, but when I build the fortran 
project using the example ex2f.F, I get a lot of LNK2019 error. I have 
tried different calling convention, but none of them works. I guess the 
there is something wrong is the fortran project.

The development environment is WIN7Pro X64 + Visual Studio.Pro 2010 + 
Intel Parallel Studio 2013XE.

The source codes and project file are attached.

Thanks and regards,

Danyang

-------- Original Message --------
Subject: 	PETSc for Windows 2.0
Date: 	Tue, 29 Oct 2013 14:22:42 +0100
From: 	MICTC <info at mic-tc.ch>
To: 	MICTC <info at mic-tc.ch>



Hello,

You were previously sent a copy of PETSc for Windows installer.

We have now created a new version which is much improved in many ways.

The new version has a new installer with a smaller footprint and it is 
much faster when installing and uninstalling.

By default it installs in the folder:

*C:\Program Files\PETSc for Windows\*

and when it unpacks it weighs about *560MB *with all the necessary dlls 
in the system folder.

Before it installs it tests if you have necessary MPI libraries 
installed or if either Visual Studio 2010 or 2012 are installed.

You can choose if you want to install a Wizard template for both 
versions or only for one of your choice.

It aborts the installer if those prerequisites are not met.

The uninstaller cleans everything up and it can be found in *Add/Remove 
Programs*.

The PETSc is the current version *3.4.2*.

The new PETSc library is built with latest Intel libraries from *Intel 
Parallel Studio XE 2013* and this time it comes with *Fortran* support 
and its built with *OpenMP* libraries.

It also comes with dlls optimized for different CPUs so the speed should 
be great whether you use old Intel or Core i5, i7 or Xeon brand.

As before, you need to install the MSMPI SDK before using the wizards:

http://www.microsoft.com/en-us/download/details.aspx?id=14737

The new installer is located here:

http://www.mic-tc.ch/downloads/PETScForWindows.zip

If you use the library please write back to us your experiences.

Best Regards,

Vladimir

*_________________________________________________________________*

*Vladimir Baros, **Projektleiter***

*Institut f?r Energietechnik (IET) & Microsoft Technical Computing 
Innovation Center*

*HSR Hochschule f?r Technik, Rapperswil*

Oberseestrasse 10

CH-8640 Rapperswil

*B?ro:*21.105 (Eichwies)

*Tel:*+41 55 2224336

*vbaros at hsr.ch <mailto:vbaros at hsr.ch>****|**www.mic-tc.ch 
<www.mictc.hsr.ch>***



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/39f3d193/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Petsc-windows-F.zip
Type: application/x-zip-compressed
Size: 22424 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/39f3d193/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Petsc-Windows-C.zip
Type: application/x-zip-compressed
Size: 3015753 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/39f3d193/attachment-0003.bin>

From jedbrown at mcs.anl.gov  Wed Oct 30 00:07:22 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 29 Oct 2013 23:07:22 -0600
Subject: [petsc-users] Bug in boundary conditions (ex25.c) ?
In-Reply-To: <CANBrw+4ocUXH6_7dRmP1uzxWsTKoowf0iR3hjs+g_HSFQU+cPA@mail.gmail.com>
References: <CANBrw+6B=D8DAEQdkahvPS9x75oUkT2zMVA5=QfZ_3raZOg5VQ@mail.gmail.com>
	<CANBrw+5jZf2p7Uz0YT5vdOMRDwyDwEfGFVDG_92fq27G9_Fv0w@mail.gmail.com>
	<87txg53ymx.fsf@mcs.anl.gov>
	<CANBrw+5Rb5_MaEauJdz7ETczfUu-cf3zoB6nbuAs8FNvu3ZGwg@mail.gmail.com>
	<87ppqp1omf.fsf@mcs.anl.gov>
	<CANBrw+4ocUXH6_7dRmP1uzxWsTKoowf0iR3hjs+g_HSFQU+cPA@mail.gmail.com>
Message-ID: <87wqkvqrcl.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
> Ok, I understand. For the moment it works well with this option.
>
> I'm still doing some tests and I would like to know if it's possible to
> view a solution in log scale ?
> To view the solution after diffusion I use
>  ierr = PetscViewerDrawOpen(PETSC_COMM_WORLD,NULL,"",0,0,300,300,&viewer);
> VecView(X,viewer);
>
> Since the values of X are large (~1e20) and vary logarithmically, I would
> like to view them in log scale. How can I do that ?

You can use VecLog() and plot that.

> Also, is it possible to view the "real coordinates" in x-axis, instead of
> the node number ?

Use DMDASetUniformCoordinates() or similar.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131029/8b748364/attachment.pgp>

From sales001 at wisdomtooling.com  Wed Oct 30 05:26:13 2013
From: sales001 at wisdomtooling.com (Mila)
Date: Wed, 30 Oct 2013 18:26:13 +0800
Subject: [petsc-users] gear tools being molded
Message-ID: <20131030182612316001717@wisdomtooling.com>

  Dear Sir or Madam,
My name is Bill from WISDOM, which is specialized in product designing , mold making, plastic & silicone parts and assembly.
Although you are super in this line, if you need any help please feel free to tell me , WISDOM entire team will do the best to quote our best price and service.
Actually, warmly welcome you visit our facility in Shenzhen China in your convenient time.
 


1) Product R&D (Included ID,MD)
2) Prototype making
3) Plastic Injection mould -Insert molding / 2K molds/ Reverse mold /Over molding / Multi-cavity hot runner molds
4) Die Cast-Aluminum & Zinc
5) Blow mould / Vaccum Mould 
6) Mass production /Assembly 
7) Machining
8) Silicone Industrial product / Silicone gift / Silicone kitchenware

Looking forward to get your response.
Have a nice day.

Best Regards
Bill 


Wisdom Industrial Group  Co., Limited   (Mold Making & Silicone Product Expert)
Tel:+86-755-2330 3009        FAx:+86-755-2330 3003 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/1452bca5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: QQ??201310070943(10-30-18-26-03).jpg
Type: image/jpeg
Size: 24596 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/1452bca5/attachment-0001.jpg>

From torquil at gmail.com  Wed Oct 30 08:58:39 2013
From: torquil at gmail.com (=?UTF-8?Q?Torquil_Macdonald_S=C3=B8rensen?=)
Date: Wed, 30 Oct 2013 14:58:39 +0100
Subject: [petsc-users] ILUTP
In-Reply-To: <CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
	<CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
Message-ID: <CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>

Thanks Matthew!

The problem I'm working on is the Dirac equation, in various number of
dimensions. I'm going to take a stab at multigrid preconditioning.

Best regards
Torquil S?rensen


On 27 October 2013 14:48, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Oct 26, 2013 at 3:05 PM, Torquil Macdonald S?rensen <
> torquil at gmail.com> wrote:
>
>> Hi!
>>
>> I have a linear problem Ax=b, with complex values, that is solved very
>> well using the ILUTP and GMRES implementations in GMM++. None of the other
>> preconditioners available in GMM++ would work (e.g. ILU and ILUT) for this
>> problem.
>>
>> Then I tried the same problem using ILU and GMRES in PETSc, with no
>> success, despite a lot of attempts at adjusting ILU settings. I always end
>> up with gigantic residual norm values. The same PETSc program works well
>> when I apply it to a different matrix A.
>>
>> I'm now suspecting that the ILU options cannot be set so as to obtain
>> ILUTP behaviour.
>>
>> What would be the recommended method to access an ILUTP preconditioner
>> from PETSc?
>>
>> According to the PETSc website, a preconditioner named ILUDT is available
>> by using the external package Hypre, but I had to deselect Hypre during the
>> PETSc build due to my use of complex numbers... So do you guys think that I
>> should transform everything to a real representation and try Hypre/ILUDT?
>>
>
> I suggest you look for a different preconditioner. The fact that you have
> so much trouble reproducing the behavior shows
> you just how fragile the performance of ILU is. It may work for a certain
> size, but fail for larger or smaller problems, or slightly
> different parameters. What problem are you solving? Usually the best
> option is to consult the literature for preconditioners
> tailored to your problem, and then reproduce them.
>
>    Matt
>
>
>> Best regards
>> Torquil S?rensen
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/e5c784d8/attachment.html>

From knepley at gmail.com  Wed Oct 30 10:48:51 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 10:48:51 -0500
Subject: [petsc-users] ILUTP
In-Reply-To: <CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
	<CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
	<CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>
Message-ID: <CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>

On Wed, Oct 30, 2013 at 8:58 AM, Torquil Macdonald S?rensen <
torquil at gmail.com> wrote:

> Thanks Matthew!
>
> The problem I'm working on is the Dirac equation, in various number of
> dimensions. I'm going to take a stab at multigrid preconditioning.
>

My limited knowledge of the Dirac Equation tells me that it is a
relativistic wave equation, and thus hyperbolic, which is quite difficult
for
MG (but doable with a bunch of work). How do you have this formulated?

  Thanks,

     Matt


> Best regards
> Torquil S?rensen
>
>
> On 27 October 2013 14:48, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sat, Oct 26, 2013 at 3:05 PM, Torquil Macdonald S?rensen <
>> torquil at gmail.com> wrote:
>>
>>> Hi!
>>>
>>> I have a linear problem Ax=b, with complex values, that is solved very
>>> well using the ILUTP and GMRES implementations in GMM++. None of the other
>>> preconditioners available in GMM++ would work (e.g. ILU and ILUT) for this
>>> problem.
>>>
>>> Then I tried the same problem using ILU and GMRES in PETSc, with no
>>> success, despite a lot of attempts at adjusting ILU settings. I always end
>>> up with gigantic residual norm values. The same PETSc program works well
>>> when I apply it to a different matrix A.
>>>
>>> I'm now suspecting that the ILU options cannot be set so as to obtain
>>> ILUTP behaviour.
>>>
>>> What would be the recommended method to access an ILUTP preconditioner
>>> from PETSc?
>>>
>>> According to the PETSc website, a preconditioner named ILUDT is
>>> available by using the external package Hypre, but I had to deselect Hypre
>>> during the PETSc build due to my use of complex numbers... So do you guys
>>> think that I should transform everything to a real representation and try
>>> Hypre/ILUDT?
>>>
>>
>> I suggest you look for a different preconditioner. The fact that you have
>> so much trouble reproducing the behavior shows
>> you just how fragile the performance of ILU is. It may work for a certain
>> size, but fail for larger or smaller problems, or slightly
>> different parameters. What problem are you solving? Usually the best
>> option is to consult the literature for preconditioners
>> tailored to your problem, and then reproduce them.
>>
>>    Matt
>>
>>
>>> Best regards
>>> Torquil S?rensen
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/31cea3ce/attachment.html>

From b.reinhardt at physik.uni-muenchen.de  Wed Oct 30 11:13:24 2013
From: b.reinhardt at physik.uni-muenchen.de (Bernhard Reinhardt)
Date: Wed, 30 Oct 2013 17:13:24 +0100
Subject: [petsc-users] DMPlex example with manual grid layout
In-Reply-To: <CAMYG4Gn8SML2f7c03p5Z5sST=L9+oh-JvhLEqemz3xj0VhHx2w@mail.gmail.com>
References: <526E47AC.2050100@physik.uni-muenchen.de>
	<CAMYG4Gn8SML2f7c03p5Z5sST=L9+oh-JvhLEqemz3xj0VhHx2w@mail.gmail.com>
Message-ID: <52713024.2070003@physik.uni-muenchen.de>

On 28.10.2013 16:43, Matthew Knepley wrote::
> On Mon, Oct 28, 2013 at 6:17 AM, Bernhard Reinhardt
> <b.reinhardt at physik.uni-muenchen.de
> <mailto:b.reinhardt at physik.uni-muenchen.de>> wrote:
>
>     Hi there,
>
>     I am trying to familiarize myself with unstructured grids. For this
>     I started with the example given in the petsc-manual (mesh of 2
>     triangles) and tried to modify it according to my needs.
>
>     However, the example seems to be incomplete. At least I was not able
>     to create a matrix without applying a DMPlexSetDimension.
>
>     In the end I decided to set up an an example as simple as possible:
>     1D transport of a scalar in a bar with periodic

aries.
>
>     Lets say I have a bar consisting of 3 volumes and 3 faces.
>     You could write this down as a vector of length 3 containing the
>     scalar in the volumes and a 3x3 matrix containing the transport
>     coefficients on the faces.
>
>     I have already tried many different ways of setting this up with
>     plex but always failed. Attached you find the version that makes
>     most sense to me. It gives me vectors of length 3 and a 3x3 matrix.
>     However, the matrix contains no writable entries and I think the
>     returned closures are not correct.
>
>     Is there a simple example with good documentation of the individual
>     steps like in the manual which is complete and working?
>
>
> I have modified your example and it is attached. Here are a few points:
>
> 1) You ask for the closure of point 4 in your Vec, and get 1.0. This is
> correct.
>
>    The closure of 4 is {4, 1, 2}, the cell + its vertices. There are
> only values on cells,
>    so you get the value on 4, which is 1.0.
>
> 2) The default for preallocation is FEM-style topology, namely that
> unknowns are connected to other
>       unknowns with overlapping support. The support of your unknowns is
> only the cell itself, so there
>       are no connections by default. I put in a call to
> DMPlexSetPreallocationCenterDimension(), which
>       is the dimension of intermediate points. In FEM these are cells,
> but in FVM they are vertices (or
>       faces). I set your topology to FVM, and you should get the
> preallocation you want.

Dear Matt,

thank you for your advice. Now I get the right allocation for my 
exemplary problem (although I have to admit I?m not entirely convinced 
for the right reasons ;-). I expanded the bar from 3 to 5 volumes, so 
that the allocation becomes clearer. See attached file.

However, I still don't get what DMPlexMatSetClosure does. What I would 
like to do, is set the operator into the matrix which computes the 
change of the value at point p. So let's say in my example the Laplacian 
operator (1, -2, 1).
E.g. Delta_p7 = A_21 p6 + A_22 p7 + A_23 p8; with A_21=1, A_22 = -2, 
A_23 = 1. However, in my example MatClosure(p7) consits only of A_22.

Probably my sieve-layout is not appropriate. I recognize that the 
Laplacian consists of 3 elements, but my volumes have only a cone size 
of 2. Nevertheless I hoped to be able to set at least A_21 and A_23.

I tried all combinations of dof=1 or dof=0 at the faces or volumes and 
PreallocationCenterDimension 0 or 1 without elucidation. I also tried 
different mesh layouts without much succes. That is

- - - - - (without any faces)
or

|-||-||-||-|-| (with double faces)
or
  _  _  _  _ _
|-||-||-||-|-| (with double faces and an additional "internal" face)

In the best case I get the foreseen allocation, but am stil am not able 
to set the right values in the matrix.

This leads me to the question: Is the sieve-layout appropriate 
(|-|-|-|-|-)? If yes, what is the envisaged way to set the operator for 
point p in the matrix?

Best regards!

Bernhard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.f90
Type: text/x-fortran
Size: 5714 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/5a2c3a83/attachment.bin>

From knepley at gmail.com  Wed Oct 30 11:35:47 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 11:35:47 -0500
Subject: [petsc-users] DMPlex example with manual grid layout
In-Reply-To: <52713024.2070003@physik.uni-muenchen.de>
References: <526E47AC.2050100@physik.uni-muenchen.de>
	<CAMYG4Gn8SML2f7c03p5Z5sST=L9+oh-JvhLEqemz3xj0VhHx2w@mail.gmail.com>
	<52713024.2070003@physik.uni-muenchen.de>
Message-ID: <CAMYG4G=oRrTjbXQg1KsOedCEyxT=615FgwmRiejdezovrbARyw@mail.gmail.com>

On Wed, Oct 30, 2013 at 11:13 AM, Bernhard Reinhardt <
b.reinhardt at physik.uni-muenchen.de> wrote:

> On 28.10.2013 16:43, Matthew Knepley wrote::
>
>> On Mon, Oct 28, 2013 at 6:17 AM, Bernhard Reinhardt
>> <b.reinhardt at physik.uni-**muenchen.de<b.reinhardt at physik.uni-muenchen.de>
>> <mailto:b.reinhardt at physik.**uni-muenchen.de<b.reinhardt at physik.uni-muenchen.de>>>
>> wrote:
>>
>>     Hi there,
>>
>>     I am trying to familiarize myself with unstructured grids. For this
>>     I started with the example given in the petsc-manual (mesh of 2
>>     triangles) and tried to modify it according to my needs.
>>
>>     However, the example seems to be incomplete. At least I was not able
>>     to create a matrix without applying a DMPlexSetDimension.
>>
>>     In the end I decided to set up an an example as simple as possible:
>>     1D transport of a scalar in a bar with periodic
>>
>
> aries.
>
>>
>>     Lets say I have a bar consisting of 3 volumes and 3 faces.
>>     You could write this down as a vector of length 3 containing the
>>     scalar in the volumes and a 3x3 matrix containing the transport
>>     coefficients on the faces.
>>
>>     I have already tried many different ways of setting this up with
>>     plex but always failed. Attached you find the version that makes
>>     most sense to me. It gives me vectors of length 3 and a 3x3 matrix.
>>     However, the matrix contains no writable entries and I think the
>>     returned closures are not correct.
>>
>>     Is there a simple example with good documentation of the individual
>>     steps like in the manual which is complete and working?
>>
>>
>> I have modified your example and it is attached. Here are a few points:
>>
>> 1) You ask for the closure of point 4 in your Vec, and get 1.0. This is
>> correct.
>>
>>    The closure of 4 is {4, 1, 2}, the cell + its vertices. There are
>> only values on cells,
>>    so you get the value on 4, which is 1.0.
>>
>> 2) The default for preallocation is FEM-style topology, namely that
>> unknowns are connected to other
>>       unknowns with overlapping support. The support of your unknowns is
>> only the cell itself, so there
>>       are no connections by default. I put in a call to
>> DMPlexSetPreallocationCenterDi**mension(), which
>>       is the dimension of intermediate points. In FEM these are cells,
>> but in FVM they are vertices (or
>>       faces). I set your topology to FVM, and you should get the
>> preallocation you want.
>>
>
> Dear Matt,
>
> thank you for your advice. Now I get the right allocation for my exemplary
> problem (although I have to admit I?m not entirely convinced for the right
> reasons ;-). I expanded the bar from 3 to 5 volumes, so that the allocation
> becomes clearer. See attached file.
>
> However, I still don't get what DMPlexMatSetClosure does.


The words used are precise. Closure means the transitive closure of a point
in the graph. For example, the
closure of point 7 is

  closure(7) = {7, 2, 3}

MatSet() takes a set of indices and a set of values and updates a Mat. Now
MatSetClosure(), first calculates the indices

  for q in closure(p):
    PetscSectionGetOffset(s, q, &off)
    PetscSectionGetDof(s, q, &dof)
    indices += [ind, ind+dof)
  MatSetValues(indices, values)


> What I would like to do, is set the operator into the matrix which
> computes the change of the value at point p. So let's say in my example the
> Laplacian operator (1, -2, 1).
> E.g. Delta_p7 = A_21 p6 + A_22 p7 + A_23 p8; with A_21=1, A_22 = -2, A_23
> = 1. However, in my example MatClosure(p7) consits only of A_22.
>

What you seem to be suggesting is FD, which you can do on a structured grid
using DMDA. On an unstructured grid, what I am familiar
with is FV, which means looping over faces instead of cells, and doing
something like MatSetStar() instead of MatSetClosure().

If you are using some unstructured method which sets whole rows of the
Jacobian at once (like FD), then you would want something like
MatSetStarClosure(), and then you would need to add the flag for ghost
cells in DMPlexDistribute(). I have never done this since all
methods I have implemented are more local.

It would help to understand what you really want to do.

   Matt


> Probably my sieve-layout is not appropriate. I recognize that the
> Laplacian consists of 3 elements, but my volumes have only a cone size of
> 2. Nevertheless I hoped to be able to set at least A_21 and A_23.
>
> I tried all combinations of dof=1 or dof=0 at the faces or volumes and
> PreallocationCenterDimension 0 or 1 without elucidation. I also tried
> different mesh layouts without much succes. That is
>
> - - - - - (without any faces)
> or
>
> |-||-||-||-|-| (with double faces)
> or
>  _  _  _  _ _
> |-||-||-||-|-| (with double faces and an additional "internal" face)
>
> In the best case I get the foreseen allocation, but am stil am not able to
> set the right values in the matrix.
>
> This leads me to the question: Is the sieve-layout appropriate
> (|-|-|-|-|-)? If yes, what is the envisaged way to set the operator for
> point p in the matrix?
>
> Best regards!
>
> Bernhard
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/a38cb788/attachment-0001.html>

From xsli at lbl.gov  Wed Oct 30 13:01:50 2013
From: xsli at lbl.gov (Xiaoye S. Li)
Date: Wed, 30 Oct 2013 11:01:50 -0700
Subject: [petsc-users] ILUTP
In-Reply-To: <CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
	<CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
	<CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>
	<CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>
Message-ID: <CAFvbobWEvYfB+X65UeE1vMN=ocCpvd0iV++y-tc_+pUVrV55pw@mail.gmail.com>

The sequential SuperLU has ILUTP implementation, not in parallel versions.
PETSc already supports the option of using SuperLU.

In SuperLU distribution:
  EXAMPLE/zitersol.c : an example to use GMRES with ILUTP preconditioner
(returned from driver SRC/zgsisx.c)
  SRC/zgsitrf.c : the actual ILUTP factorization routine

Sherry Li



On Wed, Oct 30, 2013 at 8:48 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 30, 2013 at 8:58 AM, Torquil Macdonald S?rensen <
> torquil at gmail.com> wrote:
>
>> Thanks Matthew!
>>
>> The problem I'm working on is the Dirac equation, in various number of
>> dimensions. I'm going to take a stab at multigrid preconditioning.
>>
>
> My limited knowledge of the Dirac Equation tells me that it is a
> relativistic wave equation, and thus hyperbolic, which is quite difficult
> for
> MG (but doable with a bunch of work). How do you have this formulated?
>
>   Thanks,
>
>      Matt
>
>
>> Best regards
>> Torquil S?rensen
>>
>>
>> On 27 October 2013 14:48, Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Sat, Oct 26, 2013 at 3:05 PM, Torquil Macdonald S?rensen <
>>> torquil at gmail.com> wrote:
>>>
>>>> Hi!
>>>>
>>>> I have a linear problem Ax=b, with complex values, that is solved very
>>>> well using the ILUTP and GMRES implementations in GMM++. None of the other
>>>> preconditioners available in GMM++ would work (e.g. ILU and ILUT) for this
>>>> problem.
>>>>
>>>> Then I tried the same problem using ILU and GMRES in PETSc, with no
>>>> success, despite a lot of attempts at adjusting ILU settings. I always end
>>>> up with gigantic residual norm values. The same PETSc program works well
>>>> when I apply it to a different matrix A.
>>>>
>>>> I'm now suspecting that the ILU options cannot be set so as to obtain
>>>> ILUTP behaviour.
>>>>
>>>> What would be the recommended method to access an ILUTP preconditioner
>>>> from PETSc?
>>>>
>>>> According to the PETSc website, a preconditioner named ILUDT is
>>>> available by using the external package Hypre, but I had to deselect Hypre
>>>> during the PETSc build due to my use of complex numbers... So do you guys
>>>> think that I should transform everything to a real representation and try
>>>> Hypre/ILUDT?
>>>>
>>>
>>> I suggest you look for a different preconditioner. The fact that you
>>> have so much trouble reproducing the behavior shows
>>> you just how fragile the performance of ILU is. It may work for a
>>> certain size, but fail for larger or smaller problems, or slightly
>>> different parameters. What problem are you solving? Usually the best
>>> option is to consult the literature for preconditioners
>>> tailored to your problem, and then reproduce them.
>>>
>>>    Matt
>>>
>>>
>>>> Best regards
>>>> Torquil S?rensen
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/9569f7f6/attachment.html>

From maxime.boissonneault at calculquebec.ca  Wed Oct 30 15:39:18 2013
From: maxime.boissonneault at calculquebec.ca (Maxime Boissonneault)
Date: Wed, 30 Oct 2013 16:39:18 -0400
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI 1.6.5
Message-ID: <52716E76.7080508@calculquebec.ca>

Hi,
I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI 
1.6.5 and it fails SuperLU_Dist.

I am getting the following error :

68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc 
-I/software6/mpi/openmpi/1.6.5_intel/include 
-I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64 
-fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
68304 make[1]: Leaving directory 
`/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
68305 error #13002: unexpected CFE message argument:  e. The staggered 
cosine transform may be
68306 warning #13003: message verification failed for: 556; reverting to 
internal message
68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be 
assigned to an entity of type "MPI_Request"
68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding 
Isend */
68309                                       ^
68310
68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric 
Transform has failed to release the memory.
68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding 
Isend */
68313                                         ^
68314
68315 compilation aborted for pdgstrf.c (code 1)
68316 make[1]: *** [pdgstrf.o] Error 1

Anyone has a clue on to what might be going on ?


-- 
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Qu?bec, Universit? Laval
Ph. D. en physique

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/27be7f71/attachment.html>

From gbisht at lbl.gov  Wed Oct 30 15:47:04 2013
From: gbisht at lbl.gov (Gautam Bisht)
Date: Wed, 30 Oct 2013 13:47:04 -0700
Subject: [petsc-users] Seeking an example for TS
Message-ID: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>

Hi,

I'm wondering if there is a TS example for a similar system:

d/dt [f(x)] = g(x)

For example, in PFLOTRAN, we have the following governing equation for
subsurface flow:

d/dt [ sat(P)*den(P)*por] = - g(P)

where,
sat = saturation,
den = density,
por = porosity,
P = Pressure

PFLOTRAN uses SNES to solve the nonlinear system of equations that arises
from a backward Euler time discretization. I'm trying to seek guidance on
how one would go about reformulating PFLOTRAN's equations to use TS.

Thanks,
-Gautam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/f5429eaf/attachment.html>

From knepley at gmail.com  Wed Oct 30 15:49:15 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 15:49:15 -0500
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52716E76.7080508@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
Message-ID: <CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>

On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault <
maxime.boissonneault at calculquebec.ca> wrote:

>  Hi,
> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI
> 1.6.5 and it fails SuperLU_Dist.
>
> I am getting the following error :
>
> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
> -I/software6/mpi/openmpi/1.6.5_intel/include
> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64
> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
> 68304 make[1]: Leaving directory
> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
> 68305 error #13002: unexpected CFE message argument:  e. The staggered
> cosine transform may be
> 68306 warning #13003: message verification failed for: 556; reverting to
> internal message
> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be
> assigned to an entity of type "MPI_Request"
> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding Isend
> */
> 68309                                       ^
> 68310
> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric Transform
> has failed to release the memory.
> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding Isend
> */
> 68313                                         ^
> 68314
> 68315 compilation aborted for pdgstrf.c (code 1)
> 68316 make[1]: *** [pdgstrf.o] Error 1
>
> Anyone has a clue on to what might be going on ?
>

1) Always send configure.log

2) It looks like an Intel compiler bug. Try building with --with-debugging
first

  Matt


>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/e2e997a8/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 30 15:51:57 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 14:51:57 -0600
Subject: [petsc-users] Seeking an example for TS
In-Reply-To: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>
References: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>
Message-ID: <87mwlqo51u.fsf@mcs.anl.gov>

Gautam Bisht <gbisht at lbl.gov> writes:

> Hi,
>
> I'm wondering if there is a TS example for a similar system:
>
> d/dt [f(x)] = g(x)

You can apply chain rule to produce

  f'(x) dx/dt = g(x)

and then implement TSSetIFunction and (optionally) TSSetIJacobian.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/9be63c52/attachment-0001.pgp>

From knepley at gmail.com  Wed Oct 30 15:52:11 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 15:52:11 -0500
Subject: [petsc-users] Seeking an example for TS
In-Reply-To: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>
References: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>
Message-ID: <CAMYG4G==ACDj5=RutJaDFkKaopv=pqqQM68SVrr-W75ER=8wSw@mail.gmail.com>

On Wed, Oct 30, 2013 at 3:47 PM, Gautam Bisht <gbisht at lbl.gov> wrote:

> Hi,
>
> I'm wondering if there is a TS example for a similar system:
>
> d/dt [f(x)] = g(x)
>
> For example, in PFLOTRAN, we have the following governing equation for
> subsurface flow:
>
> d/dt [ sat(P)*den(P)*por] = - g(P)
>
> where,
> sat = saturation,
> den = density,
> por = porosity,
> P = Pressure
>
> PFLOTRAN uses SNES to solve the nonlinear system of equations that arises
> from a backward Euler time discretization. I'm trying to seek guidance on
> how one would go about reformulating PFLOTRAN's equations to use TS.
>

All the TS examples (I think) have the form

  F(u, \dot u, x, t) = G(u, x, t)

where the unknowns are u, the coordinates are x, and the time is t. I would
look at the examples
(maybe ex25), and then try coding it up.

   Matt


> Thanks,
> -Gautam
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/9a2146fd/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 30 15:57:10 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 14:57:10 -0600
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52716E76.7080508@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
Message-ID: <87k3guo4t5.fsf@mcs.anl.gov>

Maxime Boissonneault <maxime.boissonneault at calculquebec.ca> writes:

> Hi,
> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI 
> 1.6.5 and it fails SuperLU_Dist.
>
> I am getting the following error :
>
> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc 
> -I/software6/mpi/openmpi/1.6.5_intel/include 
> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64 
> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
> 68304 make[1]: Leaving directory 
> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
> 68305 error #13002: unexpected CFE message argument:  e. The staggered 
> cosine transform may be
> 68306 warning #13003: message verification failed for: 556; reverting to 
> internal message
> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be 
> assigned to an entity of type "MPI_Request"
> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding 
> Isend */

This is not an error message, but the code is wrong for Open MPI (or any
MPI implementation for which MPI_Request is not typedef'd to an integer
type).

Hopefully Sherry can fix that for the next release of SuperLU_DIST.

> 68309                                       ^
> 68310
> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric 
> Transform has failed to release the memory.
> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding 
> Isend */
> 68313                                         ^
> 68314
> 68315 compilation aborted for pdgstrf.c (code 1)
> 68316 make[1]: *** [pdgstrf.o] Error 1
>
> Anyone has a clue on to what might be going on ?
>
>
> -- 
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/a9a10a97/attachment.pgp>

From knepley at gmail.com  Wed Oct 30 16:45:35 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 16:45:35 -0500
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52717CF0.7040904@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>
	<52717CF0.7040904@calculquebec.ca>
Message-ID: <CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>

On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault <
maxime.boissonneault at calculquebec.ca> wrote:

>  Hi,
> Here is the attached configure.log file.
>
> I will try the flag --with-debugging tomorrow.
>

Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2 -xHost -mkl
-fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
--with-dynamic-loading
--with-x=0 --with-mpi-compilers=1
--with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
--known-mpi-shared-libraries=1
--with-debugging=no
--with-shared-libraries=1
--with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
Don't need this --with-scalapack=1
--with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
--with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
Don't need this --with-blacs=1
--with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
--with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
--download-mumps=yes
--download-ptscotch=1
--download-superlu=yes
--download-superlu_dist=yes
--download-parmetis=yes
--download-metis=yes
--download-ml=yes
This is deprecated. Use -pc_type gamg --download-prometheus=yes
--download-hypre=yes

  Matt

>
> Maxime
>
> Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>
> On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault <
> maxime.boissonneault at calculquebec.ca> wrote:
>
>>  Hi,
>> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI
>> 1.6.5 and it fails SuperLU_Dist.
>>
>> I am getting the following error :
>>
>> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>> -I/software6/mpi/openmpi/1.6.5_intel/include
>> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64
>> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
>> 68304 make[1]: Leaving directory
>> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>> 68305 error #13002: unexpected CFE message argument:  e. The staggered
>> cosine transform may be
>> 68306 warning #13003: message verification failed for: 556; reverting to
>> internal message
>> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be
>> assigned to an entity of type "MPI_Request"
>> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>> Isend */
>> 68309                                       ^
>> 68310
>> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric Transform
>> has failed to release the memory.
>> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>> Isend */
>> 68313                                         ^
>> 68314
>> 68315 compilation aborted for pdgstrf.c (code 1)
>> 68316 make[1]: *** [pdgstrf.o] Error 1
>>
>> Anyone has a clue on to what might be going on ?
>>
>
>  1) Always send configure.log
>
>  2) It looks like an Intel compiler bug. Try building with
> --with-debugging first
>
>    Matt
>
>
>>
>> --
>> ---------------------------------
>> Maxime Boissonneault
>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>> Ph. D. en physique
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/d1e0cf54/attachment.html>

From maxime.boissonneault at calculquebec.ca  Wed Oct 30 16:48:33 2013
From: maxime.boissonneault at calculquebec.ca (Maxime Boissonneault)
Date: Wed, 30 Oct 2013 17:48:33 -0400
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
 1.6.5
In-Reply-To: <CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>
References: <52716E76.7080508@calculquebec.ca>	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>	<52717CF0.7040904@calculquebec.ca>
	<CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>
Message-ID: <52717EB1.1020706@calculquebec.ca>


Le 2013-10-30 17:45, Matthew Knepley a ?crit :
> On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault 
> <maxime.boissonneault at calculquebec.ca 
> <mailto:maxime.boissonneault at calculquebec.ca>> wrote:
>
>     Hi,
>     Here is the attached configure.log file.
>
>     I will try the flag --with-debugging tomorrow.
>
>
> Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2 -xHost 
> -mkl -fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
I had -O3 earlier. The error was the same. It is overwritten by the 
makefile anyway if you look in the details. This is not the source of 
the problem.
> --with-dynamic-loading
> --with-x=0 --with-mpi-compilers=1
> --with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
> --known-mpi-shared-libraries=1
> --with-debugging=no
> --with-shared-libraries=1
> --with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
> Don't need this --with-scalapack=1
> --with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
> --with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
> Don't need this --with-blacs=1
> --with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
> --with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
> --download-mumps=yes
> --download-ptscotch=1
> --download-superlu=yes
> --download-superlu_dist=yes
> --download-parmetis=yes
> --download-metis=yes
> --download-ml=yes
> This is deprecated. Use -pc_type gamg --download-prometheus=yes
Which part is, download-ml ? or download-hypre ? Again, this is not the 
module that fails compiling however.

Maxime
> --download-hypre=yes
>
>   Matt
>
>     Maxime
>
>     Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>>     On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault
>>     <maxime.boissonneault at calculquebec.ca
>>     <mailto:maxime.boissonneault at calculquebec.ca>> wrote:
>>
>>         Hi,
>>         I am currently trying to compile PETSc 3.4.3 with ICC 14.0
>>         and OpenMPI 1.6.5 and it fails SuperLU_Dist.
>>
>>         I am getting the following error :
>>
>>         68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>>         -I/software6/mpi/openmpi/1.6.5_intel/include
>>         -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl
>>         -fPIC -m64 -fPIC -O3 -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
>>         68304 make[1]: Leaving directory
>>         `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>>         68305 error #13002: unexpected CFE message argument:  e. The
>>         staggered cosine transform may be
>>         68306 warning #13003: message verification failed for: 556;
>>         reverting to internal message
>>         68307 pdgstrf.c(2672): warning #556: a value of type "int"
>>         cannot be assigned to an entity of type "MPI_Request"
>>         68308 U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>         Isend */
>>         68309 ^
>>         68310
>>         68311 pdgstrf.c(2672): warning #152: Fatal error:
>>         Trigonometric Transform has failed to release the memory.
>>         68312 U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>         Isend */
>>         68313 ^
>>         68314
>>         68315 compilation aborted for pdgstrf.c (code 1)
>>         68316 make[1]: *** [pdgstrf.o] Error 1
>>
>>         Anyone has a clue on to what might be going on ?
>>
>>
>>     1) Always send configure.log
>>
>>     2) It looks like an Intel compiler bug. Try building with
>>     --with-debugging first
>>
>>       Matt
>>
>>
>>         -- 
>>         ---------------------------------
>>         Maxime Boissonneault
>>         Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>         Ph. D. en physique
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>     -- 
>     ---------------------------------
>     Maxime Boissonneault
>     Analyste de calcul - Calcul Qu?bec, Universit? Laval
>     Ph. D. en physique
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Qu?bec, Universit? Laval
Ph. D. en physique

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/12e408e9/attachment-0001.html>

From knepley at gmail.com  Wed Oct 30 16:50:27 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 16:50:27 -0500
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52717EB1.1020706@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>
	<52717CF0.7040904@calculquebec.ca>
	<CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>
	<52717EB1.1020706@calculquebec.ca>
Message-ID: <CAMYG4GkPPmBTUmWDXT7P5z_hy3ftEe2p_n5ezbAJXyGvr4EqFw@mail.gmail.com>

On Wed, Oct 30, 2013 at 4:48 PM, Maxime Boissonneault <
maxime.boissonneault at calculquebec.ca> wrote:

>
> Le 2013-10-30 17:45, Matthew Knepley a ?crit :
>
> On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault <
> maxime.boissonneault at calculquebec.ca> wrote:
>
>>  Hi,
>> Here is the attached configure.log file.
>>
>> I will try the flag --with-debugging tomorrow.
>>
>
>  Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2 -xHost
> -mkl -fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
>
> I had -O3 earlier. The error was the same. It is overwritten by the
> makefile anyway if you look in the details. This is not the source of the
> problem.
>

I know, but if you leave it there with you use --with-debugging, it could
still cause the problem.


>   --with-dynamic-loading
> --with-x=0 --with-mpi-compilers=1
> --with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
>  --known-mpi-shared-libraries=1
> --with-debugging=no
> --with-shared-libraries=1
>
> --with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
> Don't need this --with-scalapack=1
>
> --with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>
> --with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
> Don't need this --with-blacs=1
>
> --with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>
> --with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
> --download-mumps=yes
> --download-ptscotch=1
> --download-superlu=yes
> --download-superlu_dist=yes
>  --download-parmetis=yes
> --download-metis=yes
> --download-ml=yes
> This is deprecated. Use -pc_type gamg --download-prometheus=yes
>
> Which part is, download-ml ? or download-hypre ? Again, this is not the
> module that fails compiling however.
>

I do not understand the question here. ML is algebraic multigrid, similar
to -pc_type gamg. Hypre has a few things in it.

  Matt


> Maxime
>
>  --download-hypre=yes
>
>    Matt
>
>>
>>  Maxime
>>
>> Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>>
>> On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault <
>> maxime.boissonneault at calculquebec.ca> wrote:
>>
>>>  Hi,
>>> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI
>>> 1.6.5 and it fails SuperLU_Dist.
>>>
>>> I am getting the following error :
>>>
>>> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>>> -I/software6/mpi/openmpi/1.6.5_intel/include
>>> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64
>>> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
>>> 68304 make[1]: Leaving directory
>>> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>>> 68305 error #13002: unexpected CFE message argument:  e. The staggered
>>> cosine transform may be
>>> 68306 warning #13003: message verification failed for: 556; reverting to
>>> internal message
>>> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be
>>> assigned to an entity of type "MPI_Request"
>>> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>> Isend */
>>> 68309                                       ^
>>> 68310
>>> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric
>>> Transform has failed to release the memory.
>>> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>> Isend */
>>> 68313                                         ^
>>> 68314
>>> 68315 compilation aborted for pdgstrf.c (code 1)
>>> 68316 make[1]: *** [pdgstrf.o] Error 1
>>>
>>> Anyone has a clue on to what might be going on ?
>>>
>>
>>  1) Always send configure.log
>>
>>  2) It looks like an Intel compiler bug. Try building with
>> --with-debugging first
>>
>>    Matt
>>
>>
>>>
>>> --
>>> ---------------------------------
>>> Maxime Boissonneault
>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>> Ph. D. en physique
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>> --
>> ---------------------------------
>> Maxime Boissonneault
>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>> Ph. D. en physique
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/9d4a58dd/attachment.html>

From maxime.boissonneault at calculquebec.ca  Wed Oct 30 16:55:01 2013
From: maxime.boissonneault at calculquebec.ca (Maxime Boissonneault)
Date: Wed, 30 Oct 2013 17:55:01 -0400
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
 1.6.5
In-Reply-To: <CAMYG4GkPPmBTUmWDXT7P5z_hy3ftEe2p_n5ezbAJXyGvr4EqFw@mail.gmail.com>
References: <52716E76.7080508@calculquebec.ca>	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>	<52717CF0.7040904@calculquebec.ca>	<CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>	<52717EB1.1020706@calculquebec.ca>
	<CAMYG4GkPPmBTUmWDXT7P5z_hy3ftEe2p_n5ezbAJXyGvr4EqFw@mail.gmail.com>
Message-ID: <52718035.6080805@calculquebec.ca>

Le 2013-10-30 17:50, Matthew Knepley a ?crit :
> On Wed, Oct 30, 2013 at 4:48 PM, Maxime Boissonneault 
> <maxime.boissonneault at calculquebec.ca 
> <mailto:maxime.boissonneault at calculquebec.ca>> wrote:
>
>
>     Le 2013-10-30 17:45, Matthew Knepley a ?crit :
>>     On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault
>>     <maxime.boissonneault at calculquebec.ca
>>     <mailto:maxime.boissonneault at calculquebec.ca>> wrote:
>>
>>         Hi,
>>         Here is the attached configure.log file.
>>
>>         I will try the flag --with-debugging tomorrow.
>>
>>
>>     Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2
>>     -xHost -mkl -fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
>     I had -O3 earlier. The error was the same. It is overwritten by
>     the makefile anyway if you look in the details. This is not the
>     source of the problem.
>
>
> I know, but if you leave it there with you use --with-debugging, it 
> could still cause the problem.
>
>>     --with-dynamic-loading
>>     --with-x=0 --with-mpi-compilers=1
>>     --with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
>>     --known-mpi-shared-libraries=1
>>     --with-debugging=no
>>     --with-shared-libraries=1
>>     --with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
>>     Don't need this --with-scalapack=1
>>     --with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>     --with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
>>     Don't need this --with-blacs=1
>>     --with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>     --with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
>>     --download-mumps=yes
>>     --download-ptscotch=1
>>     --download-superlu=yes
>>     --download-superlu_dist=yes
>>     --download-parmetis=yes
>>     --download-metis=yes
>>     --download-ml=yes
>>     This is deprecated. Use -pc_type gamg --download-prometheus=yes
>     Which part is, download-ml ? or download-hypre ? Again, this is
>     not the module that fails compiling however.
>
>
> I do not understand the question here. ML is algebraic multigrid, 
> similar to -pc_type gamg. Hypre has a few things in it.
>
Some context : I am a system administrator installing this library using 
the package list given to me by the user who will use it. I do not know 
the details of what each package does.
I do not understand which part is depecrated and what I should replace 
with -pc_type gamg.
>   Matt
>
>     Maxime
>>     --download-hypre=yes
>>
>>       Matt
>>
>>         Maxime
>>
>>         Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>>>         On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault
>>>         <maxime.boissonneault at calculquebec.ca
>>>         <mailto:maxime.boissonneault at calculquebec.ca>> wrote:
>>>
>>>             Hi,
>>>             I am currently trying to compile PETSc 3.4.3 with ICC
>>>             14.0 and OpenMPI 1.6.5 and it fails SuperLU_Dist.
>>>
>>>             I am getting the following error :
>>>
>>>             68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>>>             -I/software6/mpi/openmpi/1.6.5_intel/include
>>>             -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost
>>>             -mkl -fPIC -m64 -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c
>>>             pdgstrf.c
>>>             68304 make[1]: Leaving directory
>>>             `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>>>             68305 error #13002: unexpected CFE message argument:  e.
>>>             The staggered cosine transform may be
>>>             68306 warning #13003: message verification failed for:
>>>             556; reverting to internal message
>>>             68307 pdgstrf.c(2672): warning #556: a value of type
>>>             "int" cannot be assigned to an entity of type "MPI_Request"
>>>             68308 U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>             Isend */
>>>             68309 ^
>>>             68310
>>>             68311 pdgstrf.c(2672): warning #152: Fatal error:
>>>             Trigonometric Transform has failed to release the memory.
>>>             68312 U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>             Isend */
>>>             68313 ^
>>>             68314
>>>             68315 compilation aborted for pdgstrf.c (code 1)
>>>             68316 make[1]: *** [pdgstrf.o] Error 1
>>>
>>>             Anyone has a clue on to what might be going on ?
>>>
>>>
>>>         1) Always send configure.log
>>>
>>>         2) It looks like an Intel compiler bug. Try building with
>>>         --with-debugging first
>>>
>>>           Matt
>>>
>>>
>>>             -- 
>>>             ---------------------------------
>>>             Maxime Boissonneault
>>>             Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>>             Ph. D. en physique
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>         -- 
>>         ---------------------------------
>>         Maxime Boissonneault
>>         Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>         Ph. D. en physique
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>     -- 
>     ---------------------------------
>     Maxime Boissonneault
>     Analyste de calcul - Calcul Qu?bec, Universit? Laval
>     Ph. D. en physique
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener


-- 
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Qu?bec, Universit? Laval
Ph. D. en physique

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/1685eeb8/attachment-0001.html>

From knepley at gmail.com  Wed Oct 30 16:56:37 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 Oct 2013 16:56:37 -0500
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52718035.6080805@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>
	<52717CF0.7040904@calculquebec.ca>
	<CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>
	<52717EB1.1020706@calculquebec.ca>
	<CAMYG4GkPPmBTUmWDXT7P5z_hy3ftEe2p_n5ezbAJXyGvr4EqFw@mail.gmail.com>
	<52718035.6080805@calculquebec.ca>
Message-ID: <CAMYG4Gnd4AzQ-2ithEB+vDRAgmkStp=Q1iLVefjvLZU1f+vQTQ@mail.gmail.com>

On Wed, Oct 30, 2013 at 4:55 PM, Maxime Boissonneault <
maxime.boissonneault at calculquebec.ca> wrote:

>  Le 2013-10-30 17:50, Matthew Knepley a ?crit :
>
> On Wed, Oct 30, 2013 at 4:48 PM, Maxime Boissonneault <
> maxime.boissonneault at calculquebec.ca> wrote:
>
>>
>> Le 2013-10-30 17:45, Matthew Knepley a ?crit :
>>
>> On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault <
>> maxime.boissonneault at calculquebec.ca> wrote:
>>
>>>  Hi,
>>> Here is the attached configure.log file.
>>>
>>> I will try the flag --with-debugging tomorrow.
>>>
>>
>>  Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2 -xHost
>> -mkl -fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
>>
>> I had -O3 earlier. The error was the same. It is overwritten by the
>> makefile anyway if you look in the details. This is not the source of the
>> problem.
>>
>
>  I know, but if you leave it there with you use --with-debugging, it
> could still cause the problem.
>
>
>>   --with-dynamic-loading
>> --with-x=0 --with-mpi-compilers=1
>> --with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
>>  --known-mpi-shared-libraries=1
>> --with-debugging=no
>> --with-shared-libraries=1
>>
>> --with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
>> Don't need this --with-scalapack=1
>>
>> --with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>
>> --with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
>> Don't need this --with-blacs=1
>>
>> --with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>
>> --with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
>> --download-mumps=yes
>> --download-ptscotch=1
>> --download-superlu=yes
>> --download-superlu_dist=yes
>>  --download-parmetis=yes
>> --download-metis=yes
>> --download-ml=yes
>> This is deprecated. Use -pc_type gamg --download-prometheus=yes
>>
>> Which part is, download-ml ? or download-hypre ? Again, this is not the
>> module that fails compiling however.
>>
>
>  I do not understand the question here. ML is algebraic multigrid,
> similar to -pc_type gamg. Hypre has a few things in it.
>
>    Some context : I am a system administrator installing this library
> using the package list given to me by the user who will use it. I do not
> know the details of what each package does.
> I do not understand which part is depecrated and what I should replace
> with -pc_type gamg.
>

The package 'prometheus' is deprecated. The same functionality is now
available from within the library, so
just remove --download-prometheus.

  Thanks,

     Matt


>      Matt
>
>
>>  Maxime
>>
>>  --download-hypre=yes
>>
>>    Matt
>>
>>>
>>>  Maxime
>>>
>>> Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>>>
>>> On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault <
>>> maxime.boissonneault at calculquebec.ca> wrote:
>>>
>>>>  Hi,
>>>> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI
>>>> 1.6.5 and it fails SuperLU_Dist.
>>>>
>>>> I am getting the following error :
>>>>
>>>> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>>>> -I/software6/mpi/openmpi/1.6.5_intel/include
>>>> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64
>>>> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
>>>> 68304 make[1]: Leaving directory
>>>> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>>>> 68305 error #13002: unexpected CFE message argument:  e. The staggered
>>>> cosine transform may be
>>>> 68306 warning #13003: message verification failed for: 556; reverting
>>>> to internal message
>>>> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be
>>>> assigned to an entity of type "MPI_Request"
>>>> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>> Isend */
>>>> 68309                                       ^
>>>> 68310
>>>> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric
>>>> Transform has failed to release the memory.
>>>> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>> Isend */
>>>> 68313                                         ^
>>>> 68314
>>>> 68315 compilation aborted for pdgstrf.c (code 1)
>>>> 68316 make[1]: *** [pdgstrf.o] Error 1
>>>>
>>>> Anyone has a clue on to what might be going on ?
>>>>
>>>
>>>  1) Always send configure.log
>>>
>>>  2) It looks like an Intel compiler bug. Try building with
>>> --with-debugging first
>>>
>>>    Matt
>>>
>>>
>>>>
>>>> --
>>>> ---------------------------------
>>>> Maxime Boissonneault
>>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>>> Ph. D. en physique
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>> --
>>> ---------------------------------
>>> Maxime Boissonneault
>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>> Ph. D. en physique
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>> --
>> ---------------------------------
>> Maxime Boissonneault
>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>> Ph. D. en physique
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/43d85c51/attachment.html>

From balay at mcs.anl.gov  Wed Oct 30 18:59:49 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 Oct 2013 18:59:49 -0500 (CDT)
Subject: [petsc-users] Could not find make when building PETSc in
 Windows 7
In-Reply-To: <alpine.LFD.2.03.1310281926210.3576@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
	<1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281926210.3576@mcs.anl.gov>
Message-ID: <alpine.LFD.2.03.1310301858500.3576@mcs.anl.gov>

On Mon, 28 Oct 2013, Satish Balay wrote:

> >>>>>
>   "/usr/bin/ar: petscvecmod.o: No such file or directory
> <<<<<
> 
> Mostlikely this is related to using 'winzip' or '7zip' to untar petsc
> sources. Try using 'tar' from cygwin.
> 
> Or you can ignore this issue if not using f90 modules from your code.

Could you confirm if you used winzip or equivalent to extract petsc sources?

And does this error go away if you use 'tar -xzf petsc.tar.gz' to extract sources?

thanks,
Satish

From fd.kong at siat.ac.cn  Wed Oct 30 19:12:01 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Wed, 30 Oct 2013 18:12:01 -0600
Subject: [petsc-users] How to force the code run with a given number of
	iterations?
Message-ID: <CAN5Wd-+70yU42=yjBvoqLxd8GgCEmfXyhaakiOj9kG5H8BOiCA@mail.gmail.com>

Hi all,

How to force the code run with a given number of iterations?

I use the following script to run code, but the code stop after doing
*two*iterations:

* mpirun -n 2 ./linearElasticity  -ksp_type gmres -pc_type asm
-pc_asm_overlap 1 -sub_pc_type ilu -sub_pc_factor_levels 2 -sub_ksp_type
preonly -sub_pc_factor_mat_ordering_type rcm -ksp_view - -ksp_rtol 1e-5
-ksp_max_it 500 -ksp_monitor -ksp_monitor_singular_value*

The residual history is listed as follows:

*  0 KSP Residual norm 9.732477596672e+31
  0 KSP Residual norm 9.732477596672e+31 % max 1.000000000000e+00 min
1.000000000000e+00 max/min 1.000000000000e+00
  1 KSP Residual norm 8.532845325800e+31
  1 KSP Residual norm 8.532845325800e+31 % max 4.821161757157e+55 min
4.821161757157e+55 max/min 1.000000000000e+00
  2 KSP Residual norm 3.648279603061e+23
  2 KSP Residual norm 3.648279603061e+23 % max 1.234581012401e+79 min
2.318814661396e+55 max/min 5.324190125905e+23
*
 The residual is not small enough to stop.


Thanks,

Fande Kong,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/3daf35ca/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 30 19:38:54 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 18:38:54 -0600
Subject: [petsc-users] How to force the code run with a given number of
	iterations?
In-Reply-To: <CAN5Wd-+70yU42=yjBvoqLxd8GgCEmfXyhaakiOj9kG5H8BOiCA@mail.gmail.com>
References: <CAN5Wd-+70yU42=yjBvoqLxd8GgCEmfXyhaakiOj9kG5H8BOiCA@mail.gmail.com>
Message-ID: <877gcunujl.fsf@mcs.anl.gov>

Fande Kong <fd.kong at siat.ac.cn> writes:

> Hi all,
>
> How to force the code run with a given number of iterations?
>
> I use the following script to run code, but the code stop after doing
> *two*iterations:
>
> * mpirun -n 2 ./linearElasticity  -ksp_type gmres -pc_type asm
> -pc_asm_overlap 1 -sub_pc_type ilu -sub_pc_factor_levels 2 -sub_ksp_type
> preonly -sub_pc_factor_mat_ordering_type rcm -ksp_view - -ksp_rtol 1e-5
> -ksp_max_it 500 -ksp_monitor -ksp_monitor_singular_value*
>
> The residual history is listed as follows:
>
> *  0 KSP Residual norm 9.732477596672e+31
>   0 KSP Residual norm 9.732477596672e+31 % max 1.000000000000e+00 min
> 1.000000000000e+00 max/min 1.000000000000e+00
>   1 KSP Residual norm 8.532845325800e+31
>   1 KSP Residual norm 8.532845325800e+31 % max 4.821161757157e+55 min
> 4.821161757157e+55 max/min 1.000000000000e+00
>   2 KSP Residual norm 3.648279603061e+23
>   2 KSP Residual norm 3.648279603061e+23 % max 1.234581012401e+79 min
> 2.318814661396e+55 max/min 5.324190125905e+23
> *
>  The residual is not small enough to stop.

-ksp_converged_reason will tell you why it exited early.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/215c647e/attachment-0001.pgp>

From fd.kong at siat.ac.cn  Wed Oct 30 19:47:00 2013
From: fd.kong at siat.ac.cn (Fande Kong)
Date: Wed, 30 Oct 2013 18:47:00 -0600
Subject: [petsc-users] How to force the code run with a given number of
	iterations?
In-Reply-To: <877gcunujl.fsf@mcs.anl.gov>
References: <CAN5Wd-+70yU42=yjBvoqLxd8GgCEmfXyhaakiOj9kG5H8BOiCA@mail.gmail.com>
	<877gcunujl.fsf@mcs.anl.gov>
Message-ID: <CAN5Wd-Kmeg_e2=DZ8cnWoo0Uy1Z7h2Es2iophZuHsrUNg2rS9g@mail.gmail.com>

Thanks,

I got this message  *Linear solve converged due to CONVERGED_RTOL
iterations 2*.

What is the definition of RTOL in petsc?

I doubt the RTOL is small.*


*


On Wed, Oct 30, 2013 at 6:38 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Fande Kong <fd.kong at siat.ac.cn> writes:
>
> > Hi all,
> >
> > How to force the code run with a given number of iterations?
> >
> > I use the following script to run code, but the code stop after doing
> > *two*iterations:
> >
> > * mpirun -n 2 ./linearElasticity  -ksp_type gmres -pc_type asm
> > -pc_asm_overlap 1 -sub_pc_type ilu -sub_pc_factor_levels 2 -sub_ksp_type
> > preonly -sub_pc_factor_mat_ordering_type rcm -ksp_view - -ksp_rtol 1e-5
> > -ksp_max_it 500 -ksp_monitor -ksp_monitor_singular_value*
> >
> > The residual history is listed as follows:
> >
> > *  0 KSP Residual norm 9.732477596672e+31
> >   0 KSP Residual norm 9.732477596672e+31 % max 1.000000000000e+00 min
> > 1.000000000000e+00 max/min 1.000000000000e+00
> >   1 KSP Residual norm 8.532845325800e+31
> >   1 KSP Residual norm 8.532845325800e+31 % max 4.821161757157e+55 min
> > 4.821161757157e+55 max/min 1.000000000000e+00
> >   2 KSP Residual norm 3.648279603061e+23
> >   2 KSP Residual norm 3.648279603061e+23 % max 1.234581012401e+79 min
> > 2.318814661396e+55 max/min 5.324190125905e+23
> > *
> >  The residual is not small enough to stop.
>
> -ksp_converged_reason will tell you why it exited early.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/1aff9bbf/attachment.html>

From jedbrown at mcs.anl.gov  Wed Oct 30 19:49:31 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 18:49:31 -0600
Subject: [petsc-users] How to force the code run with a given number of
	iterations?
In-Reply-To: <CAN5Wd-Kmeg_e2=DZ8cnWoo0Uy1Z7h2Es2iophZuHsrUNg2rS9g@mail.gmail.com>
References: <CAN5Wd-+70yU42=yjBvoqLxd8GgCEmfXyhaakiOj9kG5H8BOiCA@mail.gmail.com>
	<877gcunujl.fsf@mcs.anl.gov>
	<CAN5Wd-Kmeg_e2=DZ8cnWoo0Uy1Z7h2Es2iophZuHsrUNg2rS9g@mail.gmail.com>
Message-ID: <871u32nu1w.fsf@mcs.anl.gov>

Fande Kong <fd.kong at siat.ac.cn> writes:

> Thanks,
>
> I got this message  *Linear solve converged due to CONVERGED_RTOL
> iterations 2*.
>
> What is the definition of RTOL in petsc?
>
> I doubt the RTOL is small.*
>
>
> *
>
>
> On Wed, Oct 30, 2013 at 6:38 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Fande Kong <fd.kong at siat.ac.cn> writes:
>>
>> > Hi all,
>> >
>> > How to force the code run with a given number of iterations?
>> >
>> > I use the following script to run code, but the code stop after doing
>> > *two*iterations:
>> >
>> > * mpirun -n 2 ./linearElasticity  -ksp_type gmres -pc_type asm
>> > -pc_asm_overlap 1 -sub_pc_type ilu -sub_pc_factor_levels 2 -sub_ksp_type
>> > preonly -sub_pc_factor_mat_ordering_type rcm -ksp_view - -ksp_rtol 1e-5
>> > -ksp_max_it 500 -ksp_monitor -ksp_monitor_singular_value*

You set the relative tolerance to 1e-5

>> > The residual history is listed as follows:
>> >
>> > *  0 KSP Residual norm 9.732477596672e+31

Your initial residual.

>> >   0 KSP Residual norm 9.732477596672e+31 % max 1.000000000000e+00 min
>> > 1.000000000000e+00 max/min 1.000000000000e+00
>> >   1 KSP Residual norm 8.532845325800e+31
>> >   1 KSP Residual norm 8.532845325800e+31 % max 4.821161757157e+55 min
>> > 4.821161757157e+55 max/min 1.000000000000e+00
>> >   2 KSP Residual norm 3.648279603061e+23

A residual that is 8 orders of magnitude smaller.


Your system is probably very poorly scaled (assuming taht 1e23 is not
"small" in your chosen units).  You need to fix the scaling if you want
to solve it accurately.

>> >   2 KSP Residual norm 3.648279603061e+23 % max 1.234581012401e+79 min
>> > 2.318814661396e+55 max/min 5.324190125905e+23
>> > *
>> >  The residual is not small enough to stop.
>>
>> -ksp_converged_reason will tell you why it exited early.
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/906c9705/attachment.pgp>

From jbermeol at purdue.edu  Wed Oct 30 19:57:10 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Wed, 30 Oct 2013 20:57:10 -0400 (EDT)
Subject: [petsc-users] MatMatSolve and mumps
Message-ID: <607234115.342165.1383181030220.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi, will it work if I call MatMatSolve and the solver is mumps??

Thanks

From jedbrown at mcs.anl.gov  Wed Oct 30 20:01:48 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 19:01:48 -0600
Subject: [petsc-users] [petsc-dev] MatMatSolve and mumps
In-Reply-To: <607234115.342165.1383181030220.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <607234115.342165.1383181030220.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <87y55amewz.fsf@mcs.anl.gov>

Jose David Bermeol <jbermeol at purdue.edu> writes:

> Hi, will it work if I call MatMatSolve and the solver is mumps??

Yes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/69625c0f/attachment.pgp>

From jbermeol at purdue.edu  Wed Oct 30 20:38:59 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Wed, 30 Oct 2013 21:38:59 -0400 (EDT)
Subject: [petsc-users] [petsc-dev] MatMatSolve and mumps
In-Reply-To: <87y55amewz.fsf@mcs.anl.gov>
Message-ID: <1355583275.342342.1383183539338.JavaMail.root@mailhub059.itcs.purdue.edu>

So for which solvers MatMatSolve does not work??

Thanks

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "Jose David Bermeol" <jbermeol at purdue.edu>, "petsc-dev" <petsc-dev at mcs.anl.gov>, "petsc-users" <petsc-users at mcs.anl.gov>
Sent: Wednesday, October 30, 2013 9:01:48 PM
Subject: Re: [petsc-dev] MatMatSolve and mumps

Jose David Bermeol <jbermeol at purdue.edu> writes:

> Hi, will it work if I call MatMatSolve and the solver is mumps??

Yes

From lu_qin_2000 at yahoo.com  Wed Oct 30 20:45:33 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Wed, 30 Oct 2013 18:45:33 -0700 (PDT)
Subject: [petsc-users] Could not find make when building PETSc in
	Windows 7
In-Reply-To: <alpine.LFD.2.03.1310301858500.3576@mcs.anl.gov>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
	<1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281926210.3576@mcs.anl.gov>
	<alpine.LFD.2.03.1310301858500.3576@mcs.anl.gov>
Message-ID: <1383183933.62818.YahooMailNeo@web160201.mail.bf1.yahoo.com>

Satish,

I just got chance to try your two suggestions and the errors are gone!

Many thanks,
Qin



On Wednesday, October 30, 2013 6:59 PM, Satish Balay <balay at mcs.anl.gov> wrote:
On Mon, 28 Oct 2013, Satish Balay wrote:

> >>>>>
>?  "/usr/bin/ar: petscvecmod.o: No such file or directory
> <<<<<
> 
> Mostlikely this is related to using 'winzip' or '7zip' to untar petsc
> sources. Try using 'tar' from cygwin.
> 
> Or you can ignore this issue if not using f90 modules from your code.

Could you confirm if you used winzip or equivalent to extract petsc sources?

And does this error go away if you use 'tar -xzf petsc.tar.gz' to extract sources?

thanks,

Satish


From jedbrown at mcs.anl.gov  Wed Oct 30 20:51:03 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 Oct 2013 19:51:03 -0600
Subject: [petsc-users] [petsc-dev] MatMatSolve and mumps
In-Reply-To: <1355583275.342342.1383183539338.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1355583275.342342.1383183539338.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <87mwlqmcmw.fsf@mcs.anl.gov>

Jose David Bermeol <jbermeol at purdue.edu> writes:

> So for which solvers MatMatSolve does not work??

There is a default implementation (MatMatSolve_Basic) that just solves
single vector problems in sequence.  So you can use the interface, but
the execution might vary between solvers.

The MUMPS interface is actually a liability here because they insist on
having the entire vector on rank 0.  This is a terrible interface and
not memory-scalable.  If you naively put many RHS vectors on rank 0, you
would likely run out of memory.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131030/27381edb/attachment.pgp>

From balay at mcs.anl.gov  Wed Oct 30 20:52:59 2013
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 Oct 2013 20:52:59 -0500 (CDT)
Subject: [petsc-users] Could not find make when building PETSc
 in	Windows 7
In-Reply-To: <1383183933.62818.YahooMailNeo@web160201.mail.bf1.yahoo.com>
References: <1382628823.52820.YahooMailNeo@web160203.mail.bf1.yahoo.com>
	<87a9hybs0n.fsf@mcs.anl.gov>
	<1382629408.80664.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<52694aca.ea42420a.1596.5523@mx.google.com>
	<1382632798.2753.YahooMailNeo@web160205.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310241236550.19655@mcs.anl.gov>
	<alpine.LFD.2.03.1310241404570.19655@mcs.anl.gov>
	<1382970991.50374.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281115090.3576@mcs.anl.gov>
	<1382978393.83577.YahooMailNeo@web160201.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281144030.3576@mcs.anl.gov>
	<1382996730.27474.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<alpine.LFD.2.03.1310281926210.3576@mcs.anl.gov>
	<alpine.LFD.2.03.1310301858500.3576@mcs.anl.gov>
	<1383183933.62818.YahooMailNeo@web160201.mail.bf1.yahoo.com>
Message-ID: <alpine.LFD.2.03.1310302051400.3576@mcs.anl.gov>

Thanks for the confirmation. We'll try to add in a check for this in configure.

Satish

On Wed, 30 Oct 2013, Qin Lu wrote:

> Satish,
> 
> I just got chance to try your two suggestions and the errors are gone!
> 
> Many thanks,
> Qin
> 
> 
> 
> On Wednesday, October 30, 2013 6:59 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> On Mon, 28 Oct 2013, Satish Balay wrote:
> 
> > >>>>>
> >?  "/usr/bin/ar: petscvecmod.o: No such file or directory
> > <<<<<
> > 
> > Mostlikely this is related to using 'winzip' or '7zip' to untar petsc
> > sources. Try using 'tar' from cygwin.
> > 
> > Or you can ignore this issue if not using f90 modules from your code.
> 
> Could you confirm if you used winzip or equivalent to extract petsc sources?
> 
> And does this error go away if you use 'tar -xzf petsc.tar.gz' to extract sources?
> 
> thanks,
> 
> Satish
> 
> 

From rhodri.davies at anu.edu.au  Wed Oct 30 21:19:29 2013
From: rhodri.davies at anu.edu.au (Rhodri Davies)
Date: Thu, 31 Oct 2013 02:19:29 +0000
Subject: [petsc-users] Fwd:  MatZeroRowsColumns for mpibaij
References: <5271BD65.1010507@imperial.ac.uk>
Message-ID: <BDB09259-03B6-4C62-BA5C-373F20B0243E@anu.edu.au>

Dear all,

Would you mind giving us an update on your thoughts re: this question? We're very keen to be able to use such functionally and it would be good to know if you're looking into this / have any suggestions?

Best wishes,

Rhodri

---------------------------------------------------------------------------------------------------
Research Fellow
Research School of Earth Sciences
The Australian National University
Canberra, ACT, 0200, Australia

T: +61 (0)2 6125 3643
E: rhodri.davies at anu.edu.au<mailto:rhodri.davies at anu.edu.au>
W: http://rses.anu.edu.au/people/dr-rhodri-davies
---------------------------------------------------------------------------------------------------



Begin forwarded message:

From: Stephan Kramer <s.kramer at imperial.ac.uk<mailto:s.kramer at imperial.ac.uk>>
Subject: Fwd: [petsc-users] MatZeroRowsColumns for mpibaij
Date: 31 October 2013 13:16:05 GMT+11:00
To: Rhodri Davies <rhodri.davies at anu.edu.au<mailto:rhodri.davies at anu.edu.au>>




-------- Original Message --------
Subject: [petsc-users] MatZeroRowsColumns for mpibaij
Date: Wed, 19 Jun 2013 13:22:31 +0100
From: Stephan Kramer <s.kramer at imperial.ac.uk<mailto:s.kramer at imperial.ac.uk>>
To: PETSc users list <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>

Dear all,

We have found the MatZeroRowsColumns() routine to be very useful for lifting boundary conditions. Unfortunately we also found that it isn't implemented for matrix type mpibaij and we're quite keen on
keeping the baij structure as it seems to give us significant savings for DG problems (in particular the blocked sor). I had a quick look at the implementation of MatZeroRowsColums_MPIAIJ but couldn't
really estimate how hard it would be to do something similar for mpibaij (an implementation for seqbaij exists already). Is this something that's likely to be looked at in the future?

Stephan






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/a30fd18b/attachment.html>

From christophe.ortiz at ciemat.es  Thu Oct 31 04:42:24 2013
From: christophe.ortiz at ciemat.es (Christophe Ortiz)
Date: Thu, 31 Oct 2013 10:42:24 +0100
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+4Gxp6Njv5+m2BzeYEcnL+1FbUgwWh3EV9+ZfFXcgTagg@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<9191DBF1-44A2-41CC-8809-A3D85C614984@mcs.anl.gov>
	<CANBrw+6JGO6gqxtPeegeA_E3LmCC2k47Vpf_g2bdddo3HD9Ksw@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
	<87hac8hu8y.fsf@mcs.anl.gov>
	<CANBrw+4UTHi-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>
	<87mwlzhpu4.fsf@mcs.anl.gov>
	<CANBrw+4Gxp6Njv5+m2BzeYEcnL+1FbUgwWh3EV9+ZfFXcgTagg@mail.gmail.com>
Message-ID: <CANBrw+7YZH1Bf7vd+Q=06i5zQfwMWWEfDABLwwafeuSM3z43bw@mail.gmail.com>

Hi guys,

Since I found out that with TSSetEquationType things work better I have
made some tests with a system of 2 PDEs (diff equa) and 1 ODE, with
different boundary conditions (Dirichlet and Neumann). I played around with
extreme values of the Fields and diffusion coefficients and the conclusion
is that now it works well; and I think the code is ready for the
implementation of dof >> 1.

So far I made the tests with 1 proc but with doff >> 1 I guess it will
become very time consuming and that it would be better to use several
procs. Therefore, I have few questions:

- Basically, what should I change in my code to prepare it for several
procs ?

- Should I change the matrix type ? In the current version of the code I
declared the matrix with:
  ierr = DMCreateMatrix(da,MATAIJ,&J);
Is it ok ?

- I think I should define the pattern for diagonal and offdiag blocks
  ierr = DMDASetBlockFills(da,dfill,ofill);
It's already done but I understand it only makes sense when using MPIAIJ
matrix format.

- Regarding the solver of the system, what should I change ? For the moment
I use the defaults: KSPGMRES and PCILU.

- To understand what I'm doing, when using several procs, how is actually
solved the system ? Is the whole system solved in parallel or only
IFunction and IJacobian are evaluated in parallel ?

Thanks in advance for your help.
Christophe



CIEMAT
Laboratorio Nacional de Fusi?n por Confinamiento Magn?tico
Unidad de Materiales
Edificio 2 - Planta 0 - Despacho 28m
Avenida Complutense 40,
28040 Madrid, Spain
Tel: +34 91496 2582
Fax: +34 91346 6442

--
Q
Por favor, piense en el medio ambiente antes de imprimir este mensaje.
Please consider the environment before printing this email.


On Thu, Oct 24, 2013 at 12:09 PM, Christophe Ortiz <
christophe.ortiz at ciemat.es> wrote:

>
> On Wed, Oct 23, 2013 at 7:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Christophe Ortiz <christophe.ortiz at ciemat.es> writes:
>>
>> >> That does not sound right.  Can you send a reproducible test case (best
>> >> as a patch to PETSc)?
>> >>
>> >
>> > What do you mean by patch to PETSc ? How do I do that ?
>>
>> See "contributing a small patch" (or any of the other workflows).
>>
>> https://bitbucket.org/petsc/petsc/wiki/Home
>
>
> Ok, I will read the instructions and try.
>
>
>>
>>
>> > Sorry about that. Here is the entire error message:
>> >
>> > [0]PETSC ERROR: No support for this operation for this object type!
>> > [0]PETSC ERROR: Mat type mffd!
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
>> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> > [0]PETSC ERROR: See docs/index.html for manual pages.
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
>> > mazinger.ciemat.es by u5751 Wed Oct 23 12:46:04 2013
>> > [0]PETSC ERROR: Libraries linked from
>> > /home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
>> > [0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
>> > [0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
>> > --with-cc=icc --with-debugging=1 --with-scalar-type=real
>> > --with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
>> > [0]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [0]PETSC ERROR: MatZeroEntries() line 5189 in src/mat/interface/matrix.c
>> > [0]PETSC ERROR: FormIJacobian() line 689 in
>> > src/ts/examples/tutorials/diffusion.c
>>
>> This is your code calling MatZeroEntries on the Amat (first Mat
>> argument).  The convention is to assemble into Pmat (the preconditioning
>> matrix; second Mat argument).
>>
>
> I assemble first Pmat but in my code MatZeroEntries was used with Amat,
> which PETSc did not appreciate...
> I changed that and here is the output. While without these options it
> converges--though giving solution with oscillations--now with
> -ts_arkimex_fully_implicit -snes_mf_operator it does not converge at all.
> No step is accepted, not even the first one:
>
>       TSAdapt 'basic': step   0 stage rejected t=0          + 1.000e-12
> retrying with dt=2.500e-13
>       TSAdapt 'basic': step   0 stage rejected t=0          + 2.500e-13
> retrying with dt=6.250e-14
>       TSAdapt 'basic': step   0 stage rejected t=0          + 6.250e-14
> retrying with dt=1.562e-14
>       TSAdapt 'basic': step   0 stage rejected t=0          + 1.562e-14
> retrying with dt=3.906e-15
>
> ...  ....
>       TSAdapt 'basic': step   0 stage rejected t=0          +2.888e-287
> retrying with dt=7.221e-288
>       TSAdapt 'basic': step   0 stage rejected t=0          +7.221e-288
> retrying with dt=1.805e-288
>       TSAdapt 'basic': step   0 stage rejected t=0          +1.805e-288
> retrying with dt=4.513e-289
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Vec entry at local location 1103 is not-a-number or
> infinite at end of function: Parameter number 3!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.1, Jun, 10, 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./diffusion_c on a ifort-icc-nompi-double-debug named
> mazinger.ciemat.es by u5751 Thu Oct 24 03:37:56 2013
> [0]PETSC ERROR: Libraries linked from
> /home/u5751/petsc-3.4.1/ifort-icc-nompi-double-debug/lib
> [0]PETSC ERROR: Configure run at Mon Oct 14 05:43:50 2013
> [0]PETSC ERROR: Configure options --with-mpi=0 --with-fc=ifort
> --with-cc=icc --with-debugging=1 --with-scalar-type=real
> --with-precision=double --with-blas-lapack-dir=/opt/intel/mkl
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: VecValidValues() line 30 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: SNESComputeFunction() line 1998 in
> src/snes/interface/snes.c
> [0]PETSC ERROR: SNESSolve_NEWTONLS() line 162 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 3636 in src/snes/interface/snes.c
> [0]PETSC ERROR: TSStep_ARKIMEX() line 765 in src/ts/impls/arkimex/arkimex.c
> [0]PETSC ERROR: TSStep() line 2458 in src/ts/interface/ts.c
> [0]PETSC ERROR: TSSolve() line 2583 in src/ts/interface/ts.c
> [0]PETSC ERROR: main() line 457 in src/ts/examples/tutorials/diffusion.c
> ./compile_diffusion: line 17: 23584 Aborted                 ./diffusion_c
> -ts_adapt_monitor -ts_adapt_basic_clip 0.1,1.1 -draw_pause -2
> -ts_arkimex_type 1bee -ts_max_snes_failures -1 -snes_type newtonls
> -snes_linesearch_type bt -ts_arkimex_fully_implicit -snes_mf_operator
>
> Using -snes_converged_reason and -ksp_converged_reason gives:
>
>     Linear solve did not converge due to DIVERGED_DTOL iterations 30
>   Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations
> 0
>       TSAdapt 'basic': step   0 stage rejected t=0          + 1.000e-12
> retrying with dt=2.500e-13
>
> Thus, I increased the maximum number of iterations of the KSP with
>   ierr = SNESSetMaxLinearSolveFailures(snes,100);
> and
>   ierr =
> KSPSetTolerances(ksp,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT,1000);
>
> Then it converges, steps are accepted. However, the solution obtained is
> weird, as if there was no diffusion. Moreover, values are incorrect. I
> start with a gaussian peak at 1e19 and end up with a gaussian with a peak
> at 1e25.
>
> BTW, why using -ts_arkimex_fully_implicit if all the terms are in the F
> (LHS) ? You said that by definition there is no explicit stage in that case.
>
> However, I could obtain a good solution by adding mesh points (without
> fully implicit and mf_operator). Maybe it was due to large gradients and
> not enough points.
>
> Christophe
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/f31125b8/attachment.html>

From cedric.doucet at inria.fr  Thu Oct 31 05:20:22 2013
From: cedric.doucet at inria.fr (Cedric Doucet)
Date: Thu, 31 Oct 2013 11:20:22 +0100 (CET)
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <CAMYG4G=z+FWqPkP8+eELUicvCu3eEdCF_FcZ_6bRTb7Er+2rrA@mail.gmail.com>
Message-ID: <543542989.117606.1383214822299.JavaMail.root@inria.fr>

Thank you very much for your help. 
Do you plan to add this code to Petsc? 
It would be better for us to call an official Petsc function than maintaining a Petsc-style code into our C++ software. 

Best regards, 

C?dric Doucet 

----- Mail original -----

> De: "Matthew Knepley" <knepley at gmail.com>
> ?: "Cedric Doucet" <cedric.doucet at inria.fr>
> Cc: petsc-users at mcs.anl.gov
> Envoy?: Mardi 29 Octobre 2013 17:24:03
> Objet: Re: [petsc-users] DMPlexCreateFromCellList and hybrid meshes

> On Tue, Oct 29, 2013 at 10:24 AM, Cedric Doucet <
> cedric.doucet at inria.fr > wrote:

> > Hello,
> 

> > I have a short question about the code I need to modify.
> 
> > As far as I understand, numCorner should be replaced by an array
> > numCorners[numCells] containing the number of vertices of each
> > cell.
> 
> > The body of DMPlexBuildFromCellList_Private function becomes
> 
> > ---------------------------------------------------------------------------------------------------------------------------------------
> 
> > ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
> 
> maxCorners = 0;
> > for (c = 0; c < numCells; ++c) {
> 
> > ierr = DMPlexSetConeSize(dm, c, numCorners[c]);CHKERRQ(ierr);
> 
> maxCorners = PetscMax(maxCorners, numCorners[c]);
> > }
> 
> > ierr = DMSetUp(dm);CHKERRQ(ierr);
> 

> ierr = DMGetWorkArray(dm, maxCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
> > for (c = 0, off = 0; c < numCells; ++c) {
> 

> > for (p = 0; p < numCorners[c]; ++p) {
> 
> > cone[p] = cells[off+p]+numCells;
> 
> > }
> 

> > ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
> 
> off += numCorners[c];
> > }
> 
> ierr = DMRestoreWorkArray(dm, maxCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
> > ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
> 
> > ierr = DMPlexStratify(dm);CHKERRQ(ierr);
> 

> Matt

> > ------------------------------------------------------------------------------------------------------------------------------------------
> 
> > However, I am not quite sure that DMGetWorkArray and
> > DMRestoreWorkArray should be used like this.
> 
> > I guess that DMRestoreWorkArray behaves like free function in ansi
> > c
> > but I am not sure.
> 
> > Do you think that calling DMGetWorkArray and DMRestoreWorkArray
> > inside a loop is a good thing from the point of view of
> > computational time and memory management?
> 

> > Note: the best way to do this may be to first sort cells by
> > numCorners to avoid many calls to DMGetWorkArray and
> > DMRestoreWorkArray. This is actually what I have in my own code but
> > I would like to maintain Petsc's philosophy.
> 

> > Thanks,
> 

> > C?dric
> 

> > > De: "Matthew Knepley" < knepley at gmail.com >
> > 
> 
> > > ?: "Cedric Doucet" < cedric.doucet at inria.fr >
> > 
> 
> > > Cc: petsc-users at mcs.anl.gov
> > 
> 
> > > Envoy?: Vendredi 25 Octobre 2013 16:31:59
> > 
> 
> > > Objet: Re: [petsc-users] DMPlexCreateFromCellList and hybrid
> > > meshes
> > 
> 

> > > On Fri, Oct 25, 2013 at 7:23 AM, Cedric Doucet <
> > > cedric.doucet at inria.fr > wrote:
> > 
> 

> > > > Hello,
> > > 
> > 
> 

> > > > I've noticed that DMPlexCreateFromCellList assumes that cells
> > > > have
> > > > the same number of vertices (numcorners argument).
> > > 
> > 
> 
> > > > What should be done when one wants to create a DMPlex from a
> > > > mesh
> > > > containing different types of cells?
> > > 
> > 
> 
> > > > Does one have to create several DMPlex structures and merge
> > > > them?
> > > 
> > 
> 
> > > > Does one have to create a unique DMPlex by hand?
> > > 
> > 
> 

> > > The code is very short:
> > 
> 

> > > ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
> > 
> 
> > > for (c = 0; c < numCells; ++c) {
> > 
> 
> > > ierr = DMPlexSetConeSize(dm, c, numCorners);CHKERRQ(ierr);
> > 
> 
> > > }
> > 
> 
> > > ierr = DMSetUp(dm);CHKERRQ(ierr);
> > 
> 
> > > ierr = DMGetWorkArray(dm, numCorners, PETSC_INT,
> > > &cone);CHKERRQ(ierr);
> > 
> 
> > > for (c = 0; c < numCells; ++c) {
> > 
> 
> > > for (p = 0; p < numCorners; ++p) {
> > 
> 
> > > cone[p] = cells[c*numCorners+p]+numCells;
> > 
> 
> > > }
> > 
> 
> > > ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
> > 
> 
> > > }
> > 
> 
> > > ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT,
> > > &cone);CHKERRQ(ierr);
> > 
> 
> > > ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
> > 
> 
> > > ierr = DMPlexStratify(dm);CHKERRQ(ierr);
> > 
> 

> > > This code is all in plexcreate.c. If you want different cells,
> > > you
> > > can change numCorners for each cell.
> > 
> 
> > > I could make a convenience form if necessary.
> > 
> 

> > > Matt
> > 
> 

> > > > Thank you very much for your help.
> > > 
> > 
> 

> > > > C?dric
> > > 
> > 
> 

> > > --
> > 
> 
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to
> > > which
> > > their experiments lead.
> > 
> 
> > > -- Norbert Wiener
> > 
> 
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/a01c3963/attachment-0001.html>

From knepley at gmail.com  Thu Oct 31 08:13:14 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 08:13:14 -0500
Subject: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
In-Reply-To: <543542989.117606.1383214822299.JavaMail.root@inria.fr>
References: <CAMYG4G=z+FWqPkP8+eELUicvCu3eEdCF_FcZ_6bRTb7Er+2rrA@mail.gmail.com>
	<543542989.117606.1383214822299.JavaMail.root@inria.fr>
Message-ID: <CAMYG4GnC68RQ3cJi8RzBi3PuQpqK1PyKfypH7UDgJs9gRYBSQg@mail.gmail.com>

On Thu, Oct 31, 2013 at 5:20 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:

>
> Thank you very much for your help.
> Do you plan to add this code to Petsc?
> It would be better for us to call an official Petsc function than
> maintaining a Petsc-style code into our C++ software.
>

If this is really what you need, then I will put it in. However, I intended
this to be used primarily for mesh generators, since they
tend to number cells and vertices independently and have the int and double
types hardwired.

If you are writing your own code, please consider DMPlexCreateFromDAG()
which has the flexibility for cell topology that you
want. The main differences are that it uses PETSc types and a single
contiguous numbering. Would this work for you?

  Thanks,

     Matt


> Best regards,
>
> C?dric Doucet
>
>
>
> ------------------------------
>
> *De: *"Matthew Knepley" <knepley at gmail.com>
> *?: *"Cedric Doucet" <cedric.doucet at inria.fr>
> *Cc: *petsc-users at mcs.anl.gov
> *Envoy?: *Mardi 29 Octobre 2013 17:24:03
> *Objet: *Re: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
>
> On Tue, Oct 29, 2013 at 10:24 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:
>
>> Hello,
>>
>> I have a short question about the code I need to modify.
>> As far as I understand, numCorner should be replaced by an array
>> numCorners[numCells] containing the number of vertices of each cell.
>> The body of DMPlexBuildFromCellList_Private function becomes
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------
>> ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
>>
>     maxCorners = 0;
>
>>   for (c = 0; c < numCells; ++c) {
>>     ierr = DMPlexSetConeSize(dm, c, numCorners[c]);CHKERRQ(ierr);
>>
>          maxCorners = PetscMax(maxCorners, numCorners[c]);
>
>>   }
>>   ierr = DMSetUp(dm);CHKERRQ(ierr);
>>
>
>     ierr = DMGetWorkArray(dm, maxCorners, PETSC_INT, &cone);CHKERRQ(ierr);
>
>
>>   for (c = 0, off = 0; c < numCells; ++c) {
>>     for (p = 0; p < numCorners[c]; ++p) {
>>       cone[p] = cells[off+p]+numCells;
>>     }
>>     ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
>>
>          off += numCorners[c];
>
>>    }
>>
>     ierr = DMRestoreWorkArray(dm, maxCorners, PETSC_INT,
> &cone);CHKERRQ(ierr);
>
>>
>
>>
>>   ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
>>   ierr = DMPlexStratify(dm);CHKERRQ(ierr);
>>
>
>    Matt
>
>
>>
>> ------------------------------------------------------------------------------------------------------------------------------------------
>> However, I am not quite sure that DMGetWorkArray and DMRestoreWorkArray
>> should be used like this.
>> I guess that DMRestoreWorkArray behaves like free function in ansi c but
>> I am not sure.
>> Do you think that calling DMGetWorkArray and DMRestoreWorkArray inside a
>> loop is a good thing from the point of view of computational time and
>> memory management?
>>
>> Note: the best way to do this may be to first sort cells by numCorners to
>> avoid many calls to DMGetWorkArray and DMRestoreWorkArray. This is actually
>> what I have in my own code but I would like to maintain Petsc's philosophy.
>>
>> Thanks,
>>
>> C?dric
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> *De: *"Matthew Knepley" <knepley at gmail.com>
>> *?: *"Cedric Doucet" <cedric.doucet at inria.fr>
>> *Cc: *petsc-users at mcs.anl.gov
>> *Envoy?: *Vendredi 25 Octobre 2013 16:31:59
>> *Objet: *Re: [petsc-users] DMPlexCreateFromCellList and hybrid meshes
>>
>> On Fri, Oct 25, 2013 at 7:23 AM, Cedric Doucet <cedric.doucet at inria.fr>wrote:
>>
>>> Hello,
>>>
>>> I've noticed that DMPlexCreateFromCellList assumes that cells have the
>>> same number of vertices (numcorners argument).
>>> What should be done when one wants to create a DMPlex from a mesh
>>> containing different types of cells?
>>> Does one have to create several DMPlex structures and merge them?
>>> Does one have to create a unique DMPlex by hand?
>>>
>>
>> The code is very short:
>>
>>   ierr = DMPlexSetChart(dm, 0, numCells+numVertices);CHKERRQ(ierr);
>>   for (c = 0; c < numCells; ++c) {
>>     ierr = DMPlexSetConeSize(dm, c, numCorners);CHKERRQ(ierr);
>>   }
>>   ierr = DMSetUp(dm);CHKERRQ(ierr);
>>   ierr = DMGetWorkArray(dm, numCorners, PETSC_INT, &cone);CHKERRQ(ierr);
>>   for (c = 0; c < numCells; ++c) {
>>     for (p = 0; p < numCorners; ++p) {
>>       cone[p] = cells[c*numCorners+p]+numCells;
>>     }
>>     ierr = DMPlexSetCone(dm, c, cone);CHKERRQ(ierr);
>>   }
>>   ierr = DMRestoreWorkArray(dm, numCorners, PETSC_INT,
>> &cone);CHKERRQ(ierr);
>>   ierr = DMPlexSymmetrize(dm);CHKERRQ(ierr);
>>   ierr = DMPlexStratify(dm);CHKERRQ(ierr);
>>
>> This code is all in plexcreate.c. If you want different cells, you can
>> change numCorners for each cell.
>> I could make a convenience form if necessary.
>>
>>    Matt
>>
>>
>>>
>>> Thank you very much for your help.
>>>
>>> C?dric
>>>
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/c3af5f88/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 31 08:38:24 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 07:38:24 -0600
Subject: [petsc-users] Suggestions for code with dof >> 1 ?
In-Reply-To: <CANBrw+7YZH1Bf7vd+Q=06i5zQfwMWWEfDABLwwafeuSM3z43bw@mail.gmail.com>
References: <CANBrw+6qezW28auiO2Ni=JcdqaTh_zHs5EL7nfpZtG9Suyxe6w@mail.gmail.com>
	<87hacp2tm9.fsf@mcs.anl.gov>
	<CANBrw+4_qWE9hP+_h9ZOg8Y3hp-GPneDPXxJzo_3m3wep4nh0A@mail.gmail.com>
	<CANBrw+4rUkK0-1qT-ZRvF_O9Vv6N++nbDmHBOQb7HONY+R=PHQ@mail.gmail.com>
	<87k3hepkuf.fsf@mcs.anl.gov>
	<CANBrw+74J63_=__gfU7_VmDHoi1jp1QVSpchd2ND5j_n5o5QHA@mail.gmail.com>
	<87vc0xmj66.fsf@mcs.anl.gov>
	<CANBrw+5xc10k3kDMDs6YuE9drKdcEZvdfJ9AfFAVcDTh_NZPog@mail.gmail.com>
	<CANBrw+5nzJ1bEcxMAbVqNoo31t7GgP+neufb9AgJab9ftPgncQ@mail.gmail.com>
	<CANBrw+7FfQWpBONQgUsX0ApRQfK7x+C2+FnoOV_RE4jKXsoe7w@mail.gmail.com>
	<CANBrw+64EayttkVagmQeVvp4XNODLQjOU7JZgh2JVKhm+_ym9Q@mail.gmail.com>
	<CANBrw+7MrvunXT2-XdQJje626et6272cHcxAbxQk3uYGm7sZJA@mail.gmail.com>
	<CANBrw+6hRRsODZR4bF859CX458cBqb0CbFNSMBt5xBEK3JdmGQ@mail.gmail.com>
	<CANBrw+4XEh0aeckue+u0d7zuV+tmXR_Lsx=odNzuXsJ4hpJ7WQ@mail.gmail.com>
	<87sivshwdx.fsf@mcs.anl.gov>
	<CANBrw+5i8OqpMh6YjVQLSaRLpoj3H_ZonYQXMkEZX-vdr7vv7w@mail.gmail.com>
	<87hac8hu8y.fsf@mcs.anl.gov> <CANB rw+4UTHi
	-n0MZrycERm3NMksTu63gxp2FJES0Z+1TAqDY9w@mail.gmail.com>
	<87mwlzhpu4.fsf@mcs.anl.gov>
	<CANBrw+4Gxp6Njv5+m2BzeYEcnL+1FbUgwWh3EV9+ZfFXcgTagg@mail.gmail.com>
	<CANBrw+7YZH1Bf7vd+Q=06i5zQfwMWWEfDABLwwafeuSM3z43bw@mail.gmail.com>
Message-ID: <87wqktk1bj.fsf@mcs.anl.gov>

Christophe Ortiz <christophe.ortiz at ciemat.es> writes:

> Hi guys,
>
> Since I found out that with TSSetEquationType things work better I have
> made some tests with a system of 2 PDEs (diff equa) and 1 ODE, with
> different boundary conditions (Dirichlet and Neumann). I played around with
> extreme values of the Fields and diffusion coefficients and the conclusion
> is that now it works well; and I think the code is ready for the
> implementation of dof >> 1.

Great!

> So far I made the tests with 1 proc but with doff >> 1 I guess it will
> become very time consuming and that it would be better to use several
> procs. Therefore, I have few questions:
>
> - Basically, what should I change in my code to prepare it for several
> procs ?

Just use local indexing.  The example you copied from works in parallel,
so your code might already be correct.

> - Should I change the matrix type ? In the current version of the code I
> declared the matrix with:
>   ierr = DMCreateMatrix(da,MATAIJ,&J);
> Is it ok ?

Yes, you can also pass NULL for the matrix type and the DM will choose
the type (which will be AIJ in this case).

> - I think I should define the pattern for diagonal and offdiag blocks
>   ierr = DMDASetBlockFills(da,dfill,ofill);
> It's already done but I understand it only makes sense when using MPIAIJ
> matrix format.

Yeah, and it only makes sense if the blocks are significantly sparse.
That is a less important optimization.

> - Regarding the solver of the system, what should I change ? For the moment
> I use the defaults: KSPGMRES and PCILU.

The default preconditioner in parallel is Block Jacobi with ILU.  Start
with the default and understand how it scales as you add processes, then
if it's not satisfactory look at other methods (like -pc_type gamg).

> - To understand what I'm doing, when using several procs, how is actually
> solved the system ? Is the whole system solved in parallel or only
> IFunction and IJacobian are evaluated in parallel ?

Everything is solved in parallel.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/bb59ea2c/attachment.pgp>

From info at mic-tc.ch  Thu Oct 31 09:42:06 2013
From: info at mic-tc.ch (MICTC)
Date: Thu, 31 Oct 2013 15:42:06 +0100
Subject: [petsc-users] Unresolved external symbol in PETSc for Windows
In-Reply-To: <52703E9B.7040103@gmail.com>
References: <000001ced4a9$f2e8fd00$d8baf700$@mic-tc.ch>
	<52703E9B.7040103@gmail.com>
Message-ID: <002101ced647$5f0fdbe0$1d2f93a0$@mic-tc.ch>

Hi Danyang,

 

I?m including the Fortran Visual Studio 2012 project with fixed compiler
preferences. It is using Intel Parallel Studio XE 2013.

It should work with 2010 also.

 

Regards,

Vladimir

 

From: Danyang Su [mailto:danyang.su at gmail.com] 
Sent: Wednesday, October 30, 2013 00:03
To: petsc-users
Cc: MICTC
Subject: Unresolved external symbol in PETSc for Windows

 

Hi All,

I am currently testing a PETSc fortran project for Windows. The package I
use is "PETSc for Windows 2.0" (see bellow) which can be used on windows
without installing CYGWIN on windows. I build a C project using the example
ex2.c and it works fine, but when I build the fortran project using the
example ex2f.F, I get a lot of LNK2019 error. I have tried different calling
convention, but none of them works. I guess the there is something wrong is
the fortran project. 

The development environment is WIN7Pro X64 + Visual Studio.Pro 2010 + Intel
Parallel Studio 2013XE.

The source codes and project file are attached.

Thanks and regards,

Danyang

-------- Original Message -------- 


Subject: 

PETSc for Windows 2.0


Date: 

Tue, 29 Oct 2013 14:22:42 +0100


From: 

MICTC  <mailto:info at mic-tc.ch> <info at mic-tc.ch>


To: 

MICTC  <mailto:info at mic-tc.ch> <info at mic-tc.ch>






Hello,

 

You were previously sent a copy of PETSc for Windows installer.

We have now created a new version which is much improved in many ways.

 

The new version has a new installer with a smaller footprint and it is much
faster when installing and uninstalling.

By default it installs in the folder:

C:\Program Files\PETSc for Windows\

and when it unpacks it weighs about 560MB with all the necessary dlls in the
system folder.

Before it installs it tests if you have necessary MPI libraries installed or
if either Visual Studio 2010 or 2012 are installed.

You can choose if you want to install a Wizard template for both versions or
only for one of your choice.

It aborts the installer if those prerequisites are not met.

 

The uninstaller cleans everything up and it can be found in Add/Remove
Programs.

 

The PETSc is the current version 3.4.2.

 

The new PETSc library is built with latest Intel libraries from Intel
Parallel Studio XE 2013 and this time it comes with Fortran support and its
built with OpenMP libraries.

It also comes with dlls optimized for different CPUs so the speed should be
great whether you use old Intel or Core i5, i7 or Xeon brand.

 

As before, you need to install the MSMPI SDK before using the wizards:

http://www.microsoft.com/en-us/download/details.aspx?id=14737

 

The new installer is located here:

http://www.mic-tc.ch/downloads/PETScForWindows.zip

 

 

If you use the library please write back to us your experiences.

 

 

 

Best Regards,

Vladimir

 

_________________________________________________________________

Vladimir Baros, Projektleiter

Institut f?r Energietechnik (IET) & Microsoft Technical Computing Innovation
Center

HSR Hochschule f?r Technik, Rapperswil

Oberseestrasse 10

CH-8640 Rapperswil 

B?ro: 21.105 (Eichwies)

Tel: +41 55 2224336

 <mailto:vbaros at hsr.ch> vbaros at hsr.ch | www.mic-tc.ch

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/574be435/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FortranTest.zip
Type: application/x-zip-compressed
Size: 9770 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/574be435/attachment-0001.bin>

From jbermeol at purdue.edu  Thu Oct 31 10:25:57 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Thu, 31 Oct 2013 11:25:57 -0400 (EDT)
Subject: [petsc-users] MatGetLocalSize
In-Reply-To: <183455863.344441.1383232744450.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <1176605983.344473.1383233157500.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi, I have a dense matrix of size 127200x168 over 2 MPI, so that means a have 63600 rows in on MPI process and 63600 in the other. Now When I use MatGetLocalSize in rank 0, it says that the number of local rows is 63600, but the number of local columns is 84, should not be 168?? Is this a bug??

Thanks

From jedbrown at mcs.anl.gov  Thu Oct 31 10:35:50 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 09:35:50 -0600
Subject: [petsc-users] [petsc-dev] MatGetLocalSize
In-Reply-To: <1176605983.344473.1383233157500.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <1176605983.344473.1383233157500.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <871u31jvvt.fsf@mcs.anl.gov>

Jose David Bermeol <jbermeol at purdue.edu> writes:

> Hi, I have a dense matrix of size 127200x168 over 2 MPI, so that means
> a have 63600 rows in on MPI process and 63600 in the other. Now When I
> use MatGetLocalSize in rank 0, it says that the number of local rows
> is 63600, but the number of local columns is 84, should not be 168??
> Is this a bug??

Perhaps unintuitively, the "local size" of the matrix represents the
dimensions of the vectors that the matrix can be used with, not the
ownership of entries.  Indeed, there are many matrix representations,
such as MatIS and MatElemental, where an "ownership range" cannot
describe the ownership of entries.  You can use MatGetOwnershipIS() to
get the latter (and even this is an incomplete description for MatIS).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/8e6a7e1f/attachment.pgp>

From knepley at gmail.com  Thu Oct 31 10:55:04 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 10:55:04 -0500
Subject: [petsc-users] Compiling PETSc 3.4.3 with ICC 14.0 and OpenMPI
	1.6.5
In-Reply-To: <52727B93.9010600@calculquebec.ca>
References: <52716E76.7080508@calculquebec.ca>
	<CAMYG4Gnr1_-0ZjksbOz8JnLwXc1QiQF2FjHv=6pd5S9d9Un+Dg@mail.gmail.com>
	<52717CF0.7040904@calculquebec.ca>
	<CAMYG4Gkx=oUe9B=7L7Bhxp16VVmDYc730YY9FxFkwOj27CduCw@mail.gmail.com>
	<52717EB1.1020706@calculquebec.ca>
	<CAMYG4GkPPmBTUmWDXT7P5z_hy3ftEe2p_n5ezbAJXyGvr4EqFw@mail.gmail.com>
	<52718035.6080805@calculquebec.ca>
	<CAMYG4Gnd4AzQ-2ithEB+vDRAgmkStp=Q1iLVefjvLZU1f+vQTQ@mail.gmail.com>
	<52727B93.9010600@calculquebec.ca>
Message-ID: <CAMYG4GkqfwfjDHOTPWuCG1f5B24Qfyu6UFrXEGTtSNfchcy_HA@mail.gmail.com>

On Thu, Oct 31, 2013 at 10:47 AM, Maxime Boissonneault <
maxime.boissonneault at calculquebec.ca> wrote:

>  Here is the configure.log with debugging.
>

Same compiler bug

  make[1]: Leaving directory
`/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'

error #13002: unexpected CFE message argument:  e. The staggered
cosine transform may be
warning #13003: message verification failed for: 556; reverting to
internal message

I am afraid if  you want SuperLU, you will have to revert to a non-buggy
release of ICC or use GNU.

  Thanks,

     Matt

Maxime
>
> Le 2013-10-30 17:56, Matthew Knepley a ?crit :
>
> On Wed, Oct 30, 2013 at 4:55 PM, Maxime Boissonneault <
> maxime.boissonneault at calculquebec.ca> wrote:
>
>>  Le 2013-10-30 17:50, Matthew Knepley a ?crit :
>>
>> On Wed, Oct 30, 2013 at 4:48 PM, Maxime Boissonneault <
>> maxime.boissonneault at calculquebec.ca> wrote:
>>
>>>
>>> Le 2013-10-30 17:45, Matthew Knepley a ?crit :
>>>
>>> On Wed, Oct 30, 2013 at 4:41 PM, Maxime Boissonneault <
>>> maxime.boissonneault at calculquebec.ca> wrote:
>>>
>>>>  Hi,
>>>> Here is the attached configure.log file.
>>>>
>>>> I will try the flag --with-debugging tomorrow.
>>>>
>>>
>>>  Get rid of -O2: CFLAGS="-O2 -xHost -mkl -fPIC -m64" FFLAGS="-O2 -xHost
>>> -mkl -fPIC -m64" --prefix=/software6/libs/petsc/3.4.3_intel
>>>
>>> I had -O3 earlier. The error was the same. It is overwritten by the
>>> makefile anyway if you look in the details. This is not the source of the
>>> problem.
>>>
>>
>>  I know, but if you leave it there with you use --with-debugging, it
>> could still cause the problem.
>>
>>
>>>   --with-dynamic-loading
>>> --with-x=0 --with-mpi-compilers=1
>>> --with-mpi-dir=/software6/mpi/openmpi/1.6.5_intel
>>>  --known-mpi-shared-libraries=1
>>> --with-debugging=no
>>> --with-shared-libraries=1
>>>
>>> --with-blas-lapack-dir=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64
>>> Don't need this --with-scalapack=1
>>>
>>> --with-scalapack-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>>
>>> --with-scalapack-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_scalapack_lp64.a
>>> Don't need this --with-blacs=1
>>>
>>> --with-blacs-include=/software6/compilers/intel/composer_xe_2013_sp1/include
>>>
>>> --with-blacs-lib=/software6/compilers/intel/composer_xe_2013_sp1/lib/intel64/libmkl_blacs_openmpi_lp64.a
>>> --download-mumps=yes
>>> --download-ptscotch=1
>>> --download-superlu=yes
>>> --download-superlu_dist=yes
>>>  --download-parmetis=yes
>>> --download-metis=yes
>>> --download-ml=yes
>>> This is deprecated. Use -pc_type gamg --download-prometheus=yes
>>>
>>> Which part is, download-ml ? or download-hypre ? Again, this is not the
>>> module that fails compiling however.
>>>
>>
>>  I do not understand the question here. ML is algebraic multigrid,
>> similar to -pc_type gamg. Hypre has a few things in it.
>>
>>    Some context : I am a system administrator installing this library
>> using the package list given to me by the user who will use it. I do not
>> know the details of what each package does.
>> I do not understand which part is depecrated and what I should replace
>> with -pc_type gamg.
>>
>
>  The package 'prometheus' is deprecated. The same functionality is now
> available from within the library, so
> just remove --download-prometheus.
>
>    Thanks,
>
>       Matt
>
>
>>       Matt
>>
>>
>>>  Maxime
>>>
>>>  --download-hypre=yes
>>>
>>>    Matt
>>>
>>>>
>>>>  Maxime
>>>>
>>>> Le 2013-10-30 16:49, Matthew Knepley a ?crit :
>>>>
>>>> On Wed, Oct 30, 2013 at 3:39 PM, Maxime Boissonneault <
>>>> maxime.boissonneault at calculquebec.ca> wrote:
>>>>
>>>>>  Hi,
>>>>> I am currently trying to compile PETSc 3.4.3 with ICC 14.0 and OpenMPI
>>>>> 1.6.5 and it fails SuperLU_Dist.
>>>>>
>>>>> I am getting the following error :
>>>>>
>>>>> 68303 /software6/mpi/openmpi/1.6.5_intel/bin/mpicc
>>>>> -I/software6/mpi/openmpi/1.6.5_intel/include
>>>>> -I/software6/mpi/openmpi/1.6.5_intel/include -O2 -xHost -mkl -fPIC -m64
>>>>> -fPIC -O3  -DAdd_ -DUSE_VENDOR_BLAS -c pdgstrf.c
>>>>> 68304 make[1]: Leaving directory
>>>>> `/software6/src/petsc-3.4.3-intel-try/externalpackages/SuperLU_DIST_3.3/SRC'
>>>>> 68305 error #13002: unexpected CFE message argument:  e. The staggered
>>>>> cosine transform may be
>>>>> 68306 warning #13003: message verification failed for: 556; reverting
>>>>> to internal message
>>>>> 68307 pdgstrf.c(2672): warning #556: a value of type "int" cannot be
>>>>> assigned to an entity of type "MPI_Request"
>>>>> 68308             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>>> Isend */
>>>>> 68309                                       ^
>>>>> 68310
>>>>> 68311 pdgstrf.c(2672): warning #152: Fatal error: Trigonometric
>>>>> Transform has failed to release the memory.
>>>>> 68312             U_diag_blk_send_req[krow] = 1; /* flag outstanding
>>>>> Isend */
>>>>> 68313                                         ^
>>>>> 68314
>>>>> 68315 compilation aborted for pdgstrf.c (code 1)
>>>>> 68316 make[1]: *** [pdgstrf.o] Error 1
>>>>>
>>>>> Anyone has a clue on to what might be going on ?
>>>>>
>>>>
>>>>  1) Always send configure.log
>>>>
>>>>  2) It looks like an Intel compiler bug. Try building with
>>>> --with-debugging first
>>>>
>>>>    Matt
>>>>
>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------
>>>>> Maxime Boissonneault
>>>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>>>> Ph. D. en physique
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>> --
>>>> ---------------------------------
>>>> Maxime Boissonneault
>>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>>> Ph. D. en physique
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>> --
>>> ---------------------------------
>>> Maxime Boissonneault
>>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>>> Ph. D. en physique
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>> --
>> ---------------------------------
>> Maxime Boissonneault
>> Analyste de calcul - Calcul Qu?bec, Universit? Laval
>> Ph. D. en physique
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Qu?bec, Universit? Laval
> Ph. D. en physique
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/09aafb26/attachment-0001.html>

From gbisht at lbl.gov  Thu Oct 31 11:48:28 2013
From: gbisht at lbl.gov (Gautam Bisht)
Date: Thu, 31 Oct 2013 09:48:28 -0700
Subject: [petsc-users] Seeking an example for TS
In-Reply-To: <CAMYG4G==ACDj5=RutJaDFkKaopv=pqqQM68SVrr-W75ER=8wSw@mail.gmail.com>
References: <CAPz1TncLpKvefbFL9w959a_BFq9Fh4g1SiaDzCsGq8=FRWXjJw@mail.gmail.com>
	<CAMYG4G==ACDj5=RutJaDFkKaopv=pqqQM68SVrr-W75ER=8wSw@mail.gmail.com>
Message-ID: <CAPz1Tncekv84JTF7QRY-guum0BhZoBxqBao--7qpk8RsVAsA0A@mail.gmail.com>

Thanks, Matt, Jed.

I found ex10.c useful for my case.

-Gautam


On Wed, Oct 30, 2013 at 1:52 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 30, 2013 at 3:47 PM, Gautam Bisht <gbisht at lbl.gov> wrote:
>
>> Hi,
>>
>> I'm wondering if there is a TS example for a similar system:
>>
>> d/dt [f(x)] = g(x)
>>
>> For example, in PFLOTRAN, we have the following governing equation for
>> subsurface flow:
>>
>> d/dt [ sat(P)*den(P)*por] = - g(P)
>>
>> where,
>> sat = saturation,
>> den = density,
>> por = porosity,
>> P = Pressure
>>
>> PFLOTRAN uses SNES to solve the nonlinear system of equations that arises
>> from a backward Euler time discretization. I'm trying to seek guidance on
>> how one would go about reformulating PFLOTRAN's equations to use TS.
>>
>
> All the TS examples (I think) have the form
>
>   F(u, \dot u, x, t) = G(u, x, t)
>
> where the unknowns are u, the coordinates are x, and the time is t. I
> would look at the examples
> (maybe ex25), and then try coding it up.
>
>    Matt
>
>
>> Thanks,
>> -Gautam
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/4e2e729f/attachment.html>

From bisheshkh at gmail.com  Thu Oct 31 12:02:16 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 31 Oct 2013 18:02:16 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
Message-ID: <CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>

On Tue, Oct 29, 2013 at 5:37 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 29, 2013 at 5:31 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> On Mon, Oct 28, 2013 at 9:00 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <
>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>  On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>> I would like to know if some of the petsc objects that I have
>>>>>>>>>>>>>> not used so far (IS, DMPlex, PetscSection) could be useful in the following
>>>>>>>>>>>>>> case (of irregular domains):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>>>>> The binary information of the image partitions the cube into
>>>>>>>>>>>>>> a computational domain and non-computational domain.
>>>>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>>>>> But in this matrix it should have the rows corresponding to
>>>>>>>>>>>>>> the nodes only on the computational domain. It would be nice if I can
>>>>>>>>>>>>>> easily (using (i,j,k) indexing) put on the rhs dirichlet values
>>>>>>>>>>>>>> corresponding to the boundary points.
>>>>>>>>>>>>>> Then, once the system is solved, put the values of the
>>>>>>>>>>>>>> solution back to the corresponding positions in the binary image.
>>>>>>>>>>>>>> Later, I might have to extend this for the staggered grid
>>>>>>>>>>>>>> case too.
>>>>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can
>>>>>>>>>>>>>> set up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It sounds like the right solution for this is to use
>>>>>>>>>>>>> PetscSection on top of DMDA. I am working on this, but it is really
>>>>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>>>>> development, we can help you.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>>>>> great I think.
>>>>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not
>>>>>>>>>>>> used the development version before.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Okay,
>>>>>>>>>>>
>>>>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>>>>
>>>>>>>>>>> 2) then we will need to create a PetscSection that puts unknowns
>>>>>>>>>>> where you want them
>>>>>>>>>>>
>>>>>>>>>>> 3) Setup the solver as usual
>>>>>>>>>>>
>>>>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>>>>
>>>>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the system
>>>>>>>>>> using DMDA (putting identity into the rows corresponding to the cells that
>>>>>>>>>> are not used).
>>>>>>>>>> Please let me know what I should do now.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>>>>>> partitioning cells, so you would have
>>>>>>>>> unknowns in cells. The code should look like this:
>>>>>>>>>
>>>>>>>>> PetscSectionCreate(comm, &s);
>>>>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>>>>       PetscInt point;
>>>>>>>>>
>>>>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof
>>>>>>>>> are on each vertex
>>>>>>>>>     }
>>>>>>>>>   }
>>>>>>>>> }
>>>>>>>>> PetscSectionSetUp(s);
>>>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>>>> PetscSectionDestroy(&s);
>>>>>>>>>
>>>>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I have put an example without the PetscSection here:
>>>>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>>>>> From the code you have written above, how do we let PetscSection
>>>>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>>>>> linked to?
>>>>>>>>
>>>>>>>
>>>>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>>>>
>>>>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>>>>
>>>>>>
>>>>>> Done.
>>>>>> I mask it out using isPosInDomain() function::
>>>>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(ierr
>>>>>> );
>>>>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>>>>> You know how many dof are on each vertex
>>>>>>           }
>>>>>>
>>>>>> And please let me know when I can rebuild the 'next' branch of petsc
>>>>>> so that DMDAGetCellPoint can be used. Currently compiler complains as it
>>>>>> cannot find it.
>>>>>>
>>>>>
>>>>> Pushed.
>>>>>
>>>>
>>>> Pulled it thanks, but compiler was complaining that it didn't find
>>>> DMDAGetCellPoint.
>>>> Doing grep -R 'DMDAGetCellPoint' include/
>>>> returned nothing although the implementation is there in dalocal.c
>>>> So I added the declaration to petscdmda.h file just above the
>>>> declaration of DMDAGetNumCells.
>>>>
>>>
>>> I will add the declaration.
>>>
>>>
>>>>  I have also added you as a collaborator in the github project
>>>> repository so that you can edit and add comments to the file I linked
>>>> above. My computeMatrix still uses the DMDA instead of the PetscSection, so
>>>> is it where the changes need to be made ?
>>>>
>>>
>>> Instead of MatSetValuesStencil(), you would go back to using
>>> MatSetValues(), and calculate the indices using PetscSection.
>>> You get the section
>>>
>>>   DMGetDefaultGlobalSection(dm, &gs)
>>>
>>> and then
>>>
>>>   DMDAGetCellPoint(dm, i, j, k, &p);
>>>   PetscSectionGetOffset(gs, p, &ind);
>>>   row = ind < 0 ? -(ind+1) : ind;
>>>
>>>
>>
>> I'm sorry but did not understood. For e.g., in 2D, for each point p, I
>> have 2 dof. Does PetsSectionGetOffset give me the two offsets for the dof
>> sth like ind[0] = 0 and ind[1] = 1 ?
>> How would you change the following code that sets 2D laplacian stencil
>> for 2 dof to use petscsection and Matsetvalues instead ?
>>
>
> Forget about multiple degrees of freedom until your scalar problem works.
>
>
>>  ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>     ierr = DMGetDefaultGlobalSection(dm,&gs);CHKERRQ(ierr); /*Here I get
>> the PetscSection*/
>>     for(PetscInt dof = 0; dof < 2; ++dof) {
>>         row.c = dof;
>>         for(PetscInt k = 0; k < 5; ++k) {
>>             col[k].c = dof;
>>         }
>>
>
> I can't imagine that this is what you want. You have two non-interacting
> problems.
>

I just wanted a very basic example to use PetscSection with multiple dofs.
In this example I'm solving Lap(u) = f where Lap is a component wise
laplacian and u is a vector. So if I want to apply the Lap component-wise
independently I could solve a system thrice for fx, fy and fz; but here
just to demonstrate, I wanted to have all fx, fy and fz line up in a single
vector.

I did as you suggeted to write the values in the matrix, thanks. But now is
there a way to let Petsc know automatically the size of the matrix it needs
to create for the linear system based on the PetsSection I created? When I
used only DMDA without Petscsection, this was possible, so I could just use:
    ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
    ierr =
KSPSetComputeOperators(ksp,computeMatrix2d,(void*)&testPoisson);CHKERRQ(ierr);
    ierr =
KSPSetComputeRHS(ksp,computeRhs2d,(void*)&testPoisson);CHKERRQ(ierr);
where in my computeMatrix2d function I would set the non-zero values in the
matrix.
However, now since I do not want the rows in the matrix corresponding to
the points in the grid which do not lie in the computational domain. This
means the size of the matrix will not necessarily equal the total number of
cells in DMDA. So in the following code:

 ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
    ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
    ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
    ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
    for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
        for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
            PetscInt point;
            if(isPosInDomain(&testPoisson,i,j,0)) {
                ierr = DMDAGetCellPoint(dm, i, j, 0, &point);CHKERRQ(ierr);
                ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
No. of dofs associated with the point.
            }

        }
    }
    ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
    ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
    ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);

should I myself compute proper nC (i.e. total no. of points for which I
want the matrix to have a corresponding row) before calling
PetscSectionSetChart() or is the masking out of the points inside the loop
with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
And, when you write:
PetscSectionGetOffset(gs, p, &ind);
  row = ind < 0 ? -(ind+1) : ind;

it seems you allow the possibility of getting a -ve ind when using
PetscSectionGetOffset. When would an offset to a particular dof and point?




>
>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>                 num = 0;
>>                 /*ierr =
>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>                 /*But I did now understand how I would emulate the row.c
>> and col.c info with PetscSection*/
>>                 row.i = i;  row.j = j;
>>
>
> Here you would call
>
>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>
> This means that this cell has global rows = [row, row+dof).
>
>
>>                 col[num].i = i; col[num].j = j;
>>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator for
>> only the values for only the points lying in the domain*/
>>                     v[num++] = -4;
>>                     if(isPosInDomain(ctx,i+1,j)) {
>>                         col[num].i = i+1;   col[num].j = j;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i-1,j)) {
>>                         col[num].i = i-1;   col[num].j = j;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i,j+1)) {
>>                         col[num].i = i;     col[num].j = j+1;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i,j-1)) {
>>                         col[num].i = i;     col[num].j = j-1;
>>                         v[num++] = 1;
>>                     }
>>                 } else {
>>                     v[num++] = dScale;  /*set Dirichlet identity rows for
>> points in the rectangle but outside the computational domain*/
>>                 }
>>
>
> You do the same thing for the columns, and then call
>
>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
> INSERT_VALUES);CHKERRQ(ierr);
>
>
>>                 ierr =
>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>             }
>>         }
>>     }
>>
>>
>>>     Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might be
>>>>>>>>>>>> the case. But is it always true that having identity rows in the system
>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>> reasons:
>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a point.
>>>>>>>>>>> This is just decoupling these unknowns from the rest
>>>>>>>>>>> of the problem.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>
>>>>>>>>>>>    Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/a99ad675/attachment-0001.html>

From knepley at gmail.com  Thu Oct 31 12:13:07 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 12:13:07 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
	<CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
Message-ID: <CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>

On Thu, Oct 31, 2013 at 12:02 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:

> On Tue, Oct 29, 2013 at 5:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Tue, Oct 29, 2013 at 5:31 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> On Mon, Oct 28, 2013 at 9:00 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>
>>>>>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <bisheshkh at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>>
>>>>>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <
>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <
>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>  On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>> I would like to know if some of the petsc objects that I
>>>>>>>>>>>>>>> have not used so far (IS, DMPlex, PetscSection) could be useful in the
>>>>>>>>>>>>>>> following case (of irregular domains):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>>>>>> The binary information of the image partitions the cube into
>>>>>>>>>>>>>>> a computational domain and non-computational domain.
>>>>>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>>>>>> But in this matrix it should have the rows corresponding to
>>>>>>>>>>>>>>> the nodes only on the computational domain. It would be nice if I can
>>>>>>>>>>>>>>> easily (using (i,j,k) indexing) put on the rhs dirichlet values
>>>>>>>>>>>>>>> corresponding to the boundary points.
>>>>>>>>>>>>>>> Then, once the system is solved, put the values of the
>>>>>>>>>>>>>>> solution back to the corresponding positions in the binary image.
>>>>>>>>>>>>>>> Later, I might have to extend this for the staggered grid
>>>>>>>>>>>>>>> case too.
>>>>>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I can
>>>>>>>>>>>>>>> set up the matrix with something like DMCreateMatrix ? Or what would you
>>>>>>>>>>>>>>> suggest as a suitable approach to this problem ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It sounds like the right solution for this is to use
>>>>>>>>>>>>>> PetscSection on top of DMDA. I am working on this, but it is really
>>>>>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>>>>>> development, we can help you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>>>>>> great I think.
>>>>>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not
>>>>>>>>>>>>> used the development version before.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Okay,
>>>>>>>>>>>>
>>>>>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>>>>>
>>>>>>>>>>>> 2) then we will need to create a PetscSection that puts
>>>>>>>>>>>> unknowns where you want them
>>>>>>>>>>>>
>>>>>>>>>>>> 3) Setup the solver as usual
>>>>>>>>>>>>
>>>>>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>>>>>
>>>>>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the
>>>>>>>>>>> system using DMDA (putting identity into the rows corresponding to the
>>>>>>>>>>> cells that are not used).
>>>>>>>>>>> Please let me know what I should do now.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Okay, now write a loop to setup the PetscSection. I have the DMDA
>>>>>>>>>> partitioning cells, so you would have
>>>>>>>>>> unknowns in cells. The code should look like this:
>>>>>>>>>>
>>>>>>>>>> PetscSectionCreate(comm, &s);
>>>>>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>>>>>       PetscInt point;
>>>>>>>>>>
>>>>>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many dof
>>>>>>>>>> are on each vertex
>>>>>>>>>>     }
>>>>>>>>>>   }
>>>>>>>>>> }
>>>>>>>>>> PetscSectionSetUp(s);
>>>>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>>>>> PetscSectionDestroy(&s);
>>>>>>>>>>
>>>>>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have put an example without the PetscSection here:
>>>>>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>>>>>> From the code you have written above, how do we let PetscSection
>>>>>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>>>>>> linked to?
>>>>>>>>>
>>>>>>>>
>>>>>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>>>>>
>>>>>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>>>>>
>>>>>>>
>>>>>>> Done.
>>>>>>> I mask it out using isPosInDomain() function::
>>>>>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>>>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(
>>>>>>> ierr);
>>>>>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>>>>>> You know how many dof are on each vertex
>>>>>>>           }
>>>>>>>
>>>>>>> And please let me know when I can rebuild the 'next' branch of petsc
>>>>>>> so that DMDAGetCellPoint can be used. Currently compiler complains as it
>>>>>>> cannot find it.
>>>>>>>
>>>>>>
>>>>>> Pushed.
>>>>>>
>>>>>
>>>>> Pulled it thanks, but compiler was complaining that it didn't find
>>>>> DMDAGetCellPoint.
>>>>> Doing grep -R 'DMDAGetCellPoint' include/
>>>>> returned nothing although the implementation is there in dalocal.c
>>>>> So I added the declaration to petscdmda.h file just above the
>>>>> declaration of DMDAGetNumCells.
>>>>>
>>>>
>>>> I will add the declaration.
>>>>
>>>>
>>>>>  I have also added you as a collaborator in the github project
>>>>> repository so that you can edit and add comments to the file I linked
>>>>> above. My computeMatrix still uses the DMDA instead of the PetscSection, so
>>>>> is it where the changes need to be made ?
>>>>>
>>>>
>>>> Instead of MatSetValuesStencil(), you would go back to using
>>>> MatSetValues(), and calculate the indices using PetscSection.
>>>> You get the section
>>>>
>>>>   DMGetDefaultGlobalSection(dm, &gs)
>>>>
>>>> and then
>>>>
>>>>   DMDAGetCellPoint(dm, i, j, k, &p);
>>>>   PetscSectionGetOffset(gs, p, &ind);
>>>>   row = ind < 0 ? -(ind+1) : ind;
>>>>
>>>>
>>>
>>> I'm sorry but did not understood. For e.g., in 2D, for each point p, I
>>> have 2 dof. Does PetsSectionGetOffset give me the two offsets for the dof
>>> sth like ind[0] = 0 and ind[1] = 1 ?
>>> How would you change the following code that sets 2D laplacian stencil
>>> for 2 dof to use petscsection and Matsetvalues instead ?
>>>
>>
>> Forget about multiple degrees of freedom until your scalar problem works.
>>
>>
>>>  ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>>>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>>     ierr = DMGetDefaultGlobalSection(dm,&gs);CHKERRQ(ierr); /*Here I get
>>> the PetscSection*/
>>>     for(PetscInt dof = 0; dof < 2; ++dof) {
>>>         row.c = dof;
>>>         for(PetscInt k = 0; k < 5; ++k) {
>>>             col[k].c = dof;
>>>         }
>>>
>>
>> I can't imagine that this is what you want. You have two non-interacting
>> problems.
>>
>
> I just wanted a very basic example to use PetscSection with multiple dofs.
> In this example I'm solving Lap(u) = f where Lap is a component wise
> laplacian and u is a vector. So if I want to apply the Lap component-wise
> independently I could solve a system thrice for fx, fy and fz; but here
> just to demonstrate, I wanted to have all fx, fy and fz line up in a single
> vector.
>
> I did as you suggeted to write the values in the matrix, thanks. But now
> is there a way to let Petsc know automatically the size of the matrix it
> needs to create for the linear system based on the PetsSection I created?
> When I used only DMDA without Petscsection, this was possible, so I could
> just use:
>     ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
>     ierr =
> KSPSetComputeOperators(ksp,computeMatrix2d,(void*)&testPoisson);CHKERRQ(ierr);
>     ierr =
> KSPSetComputeRHS(ksp,computeRhs2d,(void*)&testPoisson);CHKERRQ(ierr);
> where in my computeMatrix2d function I would set the non-zero values in
> the matrix.
>

DMCreateMatrix(), just as before.

   Matt


> However, now since I do not want the rows in the matrix corresponding to
> the points in the grid which do not lie in the computational domain. This
> means the size of the matrix will not necessarily equal the total number of
> cells in DMDA. So in the following code:
>
>  ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
>     ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>             PetscInt point;
>             if(isPosInDomain(&testPoisson,i,j,0)) {
>                 ierr = DMDAGetCellPoint(dm, i, j, 0, &point);CHKERRQ(ierr);
>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
> No. of dofs associated with the point.
>             }
>
>         }
>     }
>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>
> should I myself compute proper nC (i.e. total no. of points for which I
> want the matrix to have a corresponding row) before calling
> PetscSectionSetChart() or is the masking out of the points inside the loop
> with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
> And, when you write:
> PetscSectionGetOffset(gs, p, &ind);
>   row = ind < 0 ? -(ind+1) : ind;
>
> it seems you allow the possibility of getting a -ve ind when using
> PetscSectionGetOffset. When would an offset to a particular dof and point?
>
>
>
>
>>
>>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>                 num = 0;
>>>                 /*ierr =
>>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>>                 /*But I did now understand how I would emulate the row.c
>>> and col.c info with PetscSection*/
>>>                 row.i = i;  row.j = j;
>>>
>>
>> Here you would call
>>
>>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>>
>> This means that this cell has global rows = [row, row+dof).
>>
>>
>>>                  col[num].i = i; col[num].j = j;
>>>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator
>>> for only the values for only the points lying in the domain*/
>>>                     v[num++] = -4;
>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>                         col[num].i = i+1;   col[num].j = j;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>                         col[num].i = i-1;   col[num].j = j;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>                         col[num].i = i;     col[num].j = j+1;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>                         col[num].i = i;     col[num].j = j-1;
>>>                         v[num++] = 1;
>>>                     }
>>>                 } else {
>>>                     v[num++] = dScale;  /*set Dirichlet identity rows
>>> for points in the rectangle but outside the computational domain*/
>>>                 }
>>>
>>
>> You do the same thing for the columns, and then call
>>
>>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
>> INSERT_VALUES);CHKERRQ(ierr);
>>
>>
>>>                 ierr =
>>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>             }
>>>         }
>>>     }
>>>
>>>
>>>>     Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might be
>>>>>>>>>>>>> the case. But is it always true that having identity rows in the system
>>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a point.
>>>>>>>>>>>> This is just decoupling these unknowns from the rest
>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>>
>>>>>>>>>>>>    Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/01b7d425/attachment-0001.html>

From jbermeol at purdue.edu  Thu Oct 31 13:21:19 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Thu, 31 Oct 2013 14:21:19 -0400 (EDT)
Subject: [petsc-users] MatCreateDense
In-Reply-To: <456982057.345583.1383242927087.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <365213523.345706.1383243679652.JavaMail.root@mailhub059.itcs.purdue.edu>

Hi small question. In the method  MatCreateSeqDense(MPI_Comm comm,PetscInt m,PetscInt n,PetscScalar *data,Mat *A) I'm giving the local number of rows and columns, so I should pass the total number of local columns or I should pass columns/number_of_MPI process.

Thanks

From bisheshkh at gmail.com  Thu Oct 31 13:25:19 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 31 Oct 2013 19:25:19 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
	<CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
	<CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>
Message-ID: <CAEhex8i1iw+Nh7J3ECW+MTurwb-uWtw0hM_d-UnDxB8-H6LBJQ@mail.gmail.com>

On Thu, Oct 31, 2013 at 6:13 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 31, 2013 at 12:02 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> On Tue, Oct 29, 2013 at 5:37 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Tue, Oct 29, 2013 at 5:31 AM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>
>>>> On Mon, Oct 28, 2013 at 9:00 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Mon, Oct 28, 2013 at 2:51 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>>>>
>>>>>> On Mon, Oct 28, 2013 at 6:19 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>>>
>>>>>>> On Mon, Oct 28, 2013 at 10:13 AM, Bishesh Khanal <
>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>
>>>>>>>> On Mon, Oct 28, 2013 at 3:49 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Mon, Oct 28, 2013 at 9:06 AM, Bishesh Khanal <
>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Mon, Oct 28, 2013 at 1:40 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 28, 2013 at 5:30 AM, Bishesh Khanal <
>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>  On Fri, Oct 25, 2013 at 10:21 PM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 2:55 PM, Bishesh Khanal <
>>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 8:18 PM, Matthew Knepley <
>>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Oct 25, 2013 at 12:09 PM, Bishesh Khanal <
>>>>>>>>>>>>>>> bisheshkh at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>>> I would like to know if some of the petsc objects that I
>>>>>>>>>>>>>>>> have not used so far (IS, DMPlex, PetscSection) could be useful in the
>>>>>>>>>>>>>>>> following case (of irregular domains):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let's say that I have a 3D binary image (a cube).
>>>>>>>>>>>>>>>> The binary information of the image partitions the cube
>>>>>>>>>>>>>>>> into a computational domain and non-computational domain.
>>>>>>>>>>>>>>>> I must solve a pde (say a Poisson equation) only on the
>>>>>>>>>>>>>>>> computational domains (e.g: two isolated spheres within the cube). I'm
>>>>>>>>>>>>>>>> using finite difference and say a dirichlet boundary condition
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I know that I can create a dmda that will let me access the
>>>>>>>>>>>>>>>> information from this 3D binary image, get all the coefficients, rhs values
>>>>>>>>>>>>>>>> etc using the natural indexing (i,j,k).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now, I would like to create a matrix corresponding to the
>>>>>>>>>>>>>>>> laplace operator (e.g. with standard 7 pt. stencil), and the corresponding
>>>>>>>>>>>>>>>> RHS that takes care of the dirchlet values too.
>>>>>>>>>>>>>>>> But in this matrix it should have the rows corresponding to
>>>>>>>>>>>>>>>> the nodes only on the computational domain. It would be nice if I can
>>>>>>>>>>>>>>>> easily (using (i,j,k) indexing) put on the rhs dirichlet values
>>>>>>>>>>>>>>>> corresponding to the boundary points.
>>>>>>>>>>>>>>>> Then, once the system is solved, put the values of the
>>>>>>>>>>>>>>>> solution back to the corresponding positions in the binary image.
>>>>>>>>>>>>>>>> Later, I might have to extend this for the staggered grid
>>>>>>>>>>>>>>>> case too.
>>>>>>>>>>>>>>>> So is petscsection or dmplex suitable for this so that I
>>>>>>>>>>>>>>>> can set up the matrix with something like DMCreateMatrix ? Or what would
>>>>>>>>>>>>>>>> you suggest as a suitable approach to this problem ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have looked at the manual and that led me to search for a
>>>>>>>>>>>>>>>> simpler examples in petsc src directories. But most of the ones I
>>>>>>>>>>>>>>>> encountered are with FEM (and I'm not familiar at all with FEM, so these
>>>>>>>>>>>>>>>> examples serve more as a distraction with FEM jargon!)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It sounds like the right solution for this is to use
>>>>>>>>>>>>>>> PetscSection on top of DMDA. I am working on this, but it is really
>>>>>>>>>>>>>>> alpha code. If you feel comfortable with that level of
>>>>>>>>>>>>>>> development, we can help you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, with the (short) experience of using Petsc so far and
>>>>>>>>>>>>>> being familiar with the awesomeness (quick and helpful replies) of this
>>>>>>>>>>>>>> mailing list, I would like to give it a try. Please give me some pointers
>>>>>>>>>>>>>> to get going for the example case I mentioned above. A simple example of
>>>>>>>>>>>>>> using PetscSection along with DMDA for finite volume (No FEM) would be
>>>>>>>>>>>>>> great I think.
>>>>>>>>>>>>>> Just a note: I'm currently using the petsc3.4.3 and have not
>>>>>>>>>>>>>> used the development version before.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Okay,
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1)  clone the repository using Git and build the 'next' branch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) then we will need to create a PetscSection that puts
>>>>>>>>>>>>> unknowns where you want them
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3) Setup the solver as usual
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can do 1) an 3) before we do 2).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've done 1) and 3). I have one .cxx file that solves the
>>>>>>>>>>>> system using DMDA (putting identity into the rows corresponding to the
>>>>>>>>>>>> cells that are not used).
>>>>>>>>>>>> Please let me know what I should do now.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Okay, now write a loop to setup the PetscSection. I have the
>>>>>>>>>>> DMDA partitioning cells, so you would have
>>>>>>>>>>> unknowns in cells. The code should look like this:
>>>>>>>>>>>
>>>>>>>>>>> PetscSectionCreate(comm, &s);
>>>>>>>>>>> DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);
>>>>>>>>>>> PetscSectionSetChart(s, 0, nC);
>>>>>>>>>>> for (k = zs; k < zs+zm; ++k) {
>>>>>>>>>>>   for (j = ys; j < ys+ym; ++j) {
>>>>>>>>>>>     for (i = xs; i < xs+xm; ++i) {
>>>>>>>>>>>       PetscInt point;
>>>>>>>>>>>
>>>>>>>>>>>       DMDAGetCellPoint(dm, i, j, k, &point);
>>>>>>>>>>>       PetscSectionSetDof(s, point, dof); // You know how many
>>>>>>>>>>> dof are on each vertex
>>>>>>>>>>>     }
>>>>>>>>>>>   }
>>>>>>>>>>> }
>>>>>>>>>>> PetscSectionSetUp(s);
>>>>>>>>>>> DMSetDefaultSection(dm, s);
>>>>>>>>>>> PetscSectionDestroy(&s);
>>>>>>>>>>>
>>>>>>>>>>> I will merge the necessary stuff into 'next' to make this work.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have put an example without the PetscSection here:
>>>>>>>>>> https://github.com/bishesh/petscPoissonIrregular/blob/master/poissonIrregular.cxx
>>>>>>>>>> From the code you have written above, how do we let PetscSection
>>>>>>>>>> select only those cells that lie in the computational domain ?  Is it that
>>>>>>>>>> for every "point" inside the above loop, we check whether it lies in the
>>>>>>>>>> domain or not, e.g using the function isPosInDomain(...) in the .cxx file I
>>>>>>>>>> linked to?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1) Give me permission to comment on the source (I am 'knepley')
>>>>>>>>>
>>>>>>>>> 2) You mask out the (i,j,k) that you do not want in that loop
>>>>>>>>>
>>>>>>>>
>>>>>>>> Done.
>>>>>>>> I mask it out using isPosInDomain() function::
>>>>>>>>        if(isPosInDomain(&testPoisson,i,j,k)) {
>>>>>>>>             ierr = DMDAGetCellPoint(dm, i, j, k, &point);CHKERRQ(
>>>>>>>> ierr);
>>>>>>>>             ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>>>>>>>> You know how many dof are on each vertex
>>>>>>>>           }
>>>>>>>>
>>>>>>>> And please let me know when I can rebuild the 'next' branch of
>>>>>>>> petsc so that DMDAGetCellPoint can be used. Currently compiler complains as
>>>>>>>> it cannot find it.
>>>>>>>>
>>>>>>>
>>>>>>> Pushed.
>>>>>>>
>>>>>>
>>>>>> Pulled it thanks, but compiler was complaining that it didn't find
>>>>>> DMDAGetCellPoint.
>>>>>> Doing grep -R 'DMDAGetCellPoint' include/
>>>>>> returned nothing although the implementation is there in dalocal.c
>>>>>> So I added the declaration to petscdmda.h file just above the
>>>>>> declaration of DMDAGetNumCells.
>>>>>>
>>>>>
>>>>> I will add the declaration.
>>>>>
>>>>>
>>>>>>  I have also added you as a collaborator in the github project
>>>>>> repository so that you can edit and add comments to the file I linked
>>>>>> above. My computeMatrix still uses the DMDA instead of the PetscSection, so
>>>>>> is it where the changes need to be made ?
>>>>>>
>>>>>
>>>>> Instead of MatSetValuesStencil(), you would go back to using
>>>>> MatSetValues(), and calculate the indices using PetscSection.
>>>>> You get the section
>>>>>
>>>>>   DMGetDefaultGlobalSection(dm, &gs)
>>>>>
>>>>> and then
>>>>>
>>>>>   DMDAGetCellPoint(dm, i, j, k, &p);
>>>>>   PetscSectionGetOffset(gs, p, &ind);
>>>>>   row = ind < 0 ? -(ind+1) : ind;
>>>>>
>>>>>
>>>>
>>>> I'm sorry but did not understood. For e.g., in 2D, for each point p, I
>>>> have 2 dof. Does PetsSectionGetOffset give me the two offsets for the dof
>>>> sth like ind[0] = 0 and ind[1] = 1 ?
>>>> How would you change the following code that sets 2D laplacian stencil
>>>> for 2 dof to use petscsection and Matsetvalues instead ?
>>>>
>>>
>>> Forget about multiple degrees of freedom until your scalar problem works.
>>>
>>>
>>>>  ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>>>>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>>>     ierr = DMGetDefaultGlobalSection(dm,&gs);CHKERRQ(ierr); /*Here I
>>>> get the PetscSection*/
>>>>     for(PetscInt dof = 0; dof < 2; ++dof) {
>>>>         row.c = dof;
>>>>         for(PetscInt k = 0; k < 5; ++k) {
>>>>             col[k].c = dof;
>>>>         }
>>>>
>>>
>>> I can't imagine that this is what you want. You have two non-interacting
>>> problems.
>>>
>>
>> I just wanted a very basic example to use PetscSection with multiple
>> dofs. In this example I'm solving Lap(u) = f where Lap is a component wise
>> laplacian and u is a vector. So if I want to apply the Lap component-wise
>> independently I could solve a system thrice for fx, fy and fz; but here
>> just to demonstrate, I wanted to have all fx, fy and fz line up in a single
>> vector.
>>
>> I did as you suggeted to write the values in the matrix, thanks. But now
>> is there a way to let Petsc know automatically the size of the matrix it
>> needs to create for the linear system based on the PetsSection I created?
>> When I used only DMDA without Petscsection, this was possible, so I could
>> just use:
>>     ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
>>     ierr =
>> KSPSetComputeOperators(ksp,computeMatrix2d,(void*)&testPoisson);CHKERRQ(ierr);
>>     ierr =
>> KSPSetComputeRHS(ksp,computeRhs2d,(void*)&testPoisson);CHKERRQ(ierr);
>> where in my computeMatrix2d function I would set the non-zero values in
>> the matrix.
>>
>
> DMCreateMatrix(), just as before.
>

I did not call DMCreateMatrix() before when using just dmda (and it is
working). In any case, I added a call to DMCreateMatrix() now but still
there are problems. Here are the code snippets and the error I get:

//Setting up the section and linking it to DM:
ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
    ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
    ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
    for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
        for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
            PetscInt point;
            if(isPosInDomain(&testPoisson,i,j,0)) {
                ierr = DMDAGetCellPoint(dm, i, j, 0, &point);CHKERRQ(ierr);
                ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
No. of dofs associated with the point.
            }
        }
    }
    ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
    ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
    ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
    ierr = DMCreateMatrix(dm,&A);CHKERRQ(ierr);

 //Set up KSP:
    ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
    ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
    ierr =
KSPSetComputeOperators(ksp,computeMatrix2dSection,(void*)&testPoisson);CHKERRQ(ierr);
    ierr =
KSPSetComputeRHS(ksp,computeRhs2dSection,(void*)&testPoisson);CHKERRQ(ierr);
    ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
    ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);

------------------------------------------------------
The computeMatrix2dSection function has:

ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
    ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
    ierr = DMGetDefaultGlobalSection(da,&gs);CHKERRQ(ierr);
    for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
        for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
            if (isPosInDomain(ctx,i,j)) {
                ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
                ierr = PetscSectionGetOffset(gs,point,&row);
                ierr = PetscSectionGetDof(gs,point,&dof);
                for(PetscInt cDof = 0; cDof < dof; ++cDof) {
                    num = 0;
                    row+=cDof;
                    col[num] = row;         //(i,j) position
                    v[num++] = -4;
                    if(isPosInDomain(ctx,i+1,j)) {
                        ierr =
DMDAGetCellPoint(da,i+1,j,0,&point);CHKERRQ(ierr);
                        ierr = PetscSectionGetOffset(gs,point,&col[num]);
                        col[num] += cDof;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i-1,j)) {
                        ierr =
DMDAGetCellPoint(da,i-1,j,0,&point);CHKERRQ(ierr);
                        ierr = PetscSectionGetOffset(gs,point,&col[num]);
                        col[num] += cDof;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i,j+1)) {
                        ierr =
DMDAGetCellPoint(da,i,j+1,0,&point);CHKERRQ(ierr);
                        ierr = PetscSectionGetOffset(gs,point,&col[num]);
                        col[num] += cDof;
                        v[num++] = 1;
                    }
                    if(isPosInDomain(ctx,i,j-1)) {
                        ierr =
DMDAGetCellPoint(da,i,j-1,0,&point);CHKERRQ(ierr);
                        ierr = PetscSectionGetOffset(gs,point,&col[num]);
                        col[num] += cDof;
                        v[num++] = 1;
                    }
                    ierr =
MatSetValues(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
                }
            }
        }
    }
    ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
    ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);

But this is not working. For e.g. for a 6X6 grid size I get the following
error:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: New nonzero at (0,6) caused a malloc!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision:
61e5e40bb2c5bf2423e94b71a15fef47e411b0da  GIT Date: 2013-10-25 21:47:45
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: build/poissonIrregular on a arch-linux2-cxx-debug named
edwards by bkhanal Thu Oct 31 19:23:58 2013
[0]PETSC ERROR: Libraries linked from
/home/bkhanal/Documents/softwares/petsc/arch-linux2-cxx-debug/lib
[0]PETSC ERROR: Configure run at Sat Oct 26 16:35:15 2013
[0]PETSC ERROR: Configure options --download-mpich
-download-f-blas-lapack=1 --download-metis --download-parmetis
--download-superlu_dist --download-scalapack --download-mumps
--download-hypre --with-clanguage=cxx
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatSetValues_SeqAIJ() line 413 in
src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: MatSetValues() line 1130 in src/mat/interface/matrix.c
[0]PETSC ERROR: computeMatrix2dSection() line 318 in
/user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
[0]PETSC ERROR: KSPSetUp() line 228 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: main() line 598 in
/user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0


>    Matt
>
>
>> However, now since I do not want the rows in the matrix corresponding to
>> the points in the grid which do not lie in the computational domain. This
>> means the size of the matrix will not necessarily equal the total number of
>> cells in DMDA. So in the following code:
>>
>>  ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
>>     ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>             PetscInt point;
>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>> &point);CHKERRQ(ierr);
>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>> No. of dofs associated with the point.
>>             }
>>
>>         }
>>     }
>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>
>> should I myself compute proper nC (i.e. total no. of points for which I
>> want the matrix to have a corresponding row) before calling
>> PetscSectionSetChart() or is the masking out of the points inside the loop
>> with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
>> And, when you write:
>> PetscSectionGetOffset(gs, p, &ind);
>>   row = ind < 0 ? -(ind+1) : ind;
>>
>> it seems you allow the possibility of getting a -ve ind when using
>> PetscSectionGetOffset. When would an offset to a particular dof and point?
>>
>>
>>
>>
>>>
>>>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>>                 num = 0;
>>>>                 /*ierr =
>>>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>>>                 /*But I did now understand how I would emulate the
>>>> row.c and col.c info with PetscSection*/
>>>>                 row.i = i;  row.j = j;
>>>>
>>>
>>> Here you would call
>>>
>>>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>>>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>>>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>>>
>>> This means that this cell has global rows = [row, row+dof).
>>>
>>>
>>>>                  col[num].i = i; col[num].j = j;
>>>>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator
>>>> for only the values for only the points lying in the domain*/
>>>>                     v[num++] = -4;
>>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>>                         col[num].i = i+1;   col[num].j = j;
>>>>                         v[num++] = 1;
>>>>                     }
>>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>>                         col[num].i = i-1;   col[num].j = j;
>>>>                         v[num++] = 1;
>>>>                     }
>>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>>                         col[num].i = i;     col[num].j = j+1;
>>>>                         v[num++] = 1;
>>>>                     }
>>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>>                         col[num].i = i;     col[num].j = j-1;
>>>>                         v[num++] = 1;
>>>>                     }
>>>>                 } else {
>>>>                     v[num++] = dScale;  /*set Dirichlet identity rows
>>>> for points in the rectangle but outside the computational domain*/
>>>>                 }
>>>>
>>>
>>> You do the same thing for the columns, and then call
>>>
>>>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
>>> INSERT_VALUES);CHKERRQ(ierr);
>>>
>>>
>>>>                 ierr =
>>>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>>             }
>>>>         }
>>>>     }
>>>>
>>>>
>>>>>     Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>    Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might be
>>>>>>>>>>>>>> the case. But is it always true that having identity rows in the system
>>>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a
>>>>>>>>>>>>> point. This is just decoupling these unknowns from the rest
>>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/0d83aaca/attachment-0001.html>

From knepley at gmail.com  Thu Oct 31 13:37:43 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 13:37:43 -0500
Subject: [petsc-users] [petsc-dev] MatCreateDense
In-Reply-To: <365213523.345706.1383243679652.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <456982057.345583.1383242927087.JavaMail.root@mailhub059.itcs.purdue.edu>
	<365213523.345706.1383243679652.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4GmwCv4Rv11eJ-dEF0NmOGXO_zKn8=b1yv45UTBqkLMWdg@mail.gmail.com>

On Thu, Oct 31, 2013 at 1:21 PM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> Hi small question. In the method  MatCreateSeqDense(MPI_Comm comm,PetscInt
> m,PetscInt n,PetscScalar *data,Mat *A) I'm giving the local number of rows
> and columns, so I should pass the total number of local columns or I should
> pass columns/number_of_MPI process.
>

If its sequential, you pass the total number of columns. If its
MatCreateMPIDense(), pass the number of local columns. This
is used to create a vector with the same layout as y = A^T x.

  Matt


> Thanks
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/bb85dc36/attachment.html>

From knepley at gmail.com  Thu Oct 31 13:43:05 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 13:43:05 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8i1iw+Nh7J3ECW+MTurwb-uWtw0hM_d-UnDxB8-H6LBJQ@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
	<CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
	<CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>
	<CAEhex8i1iw+Nh7J3ECW+MTurwb-uWtw0hM_d-UnDxB8-H6LBJQ@mail.gmail.com>
Message-ID: <CAMYG4GkDQLCXmWEcDJKZNSUnHbCpUicr0tr5cqaUxmuUYYWyVA@mail.gmail.com>

On Thu, Oct 31, 2013 at 1:25 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

> I did not call DMCreateMatrix() before when using just dmda (and it is
> working). In any case, I added a call to DMCreateMatrix() now but still
> there are problems. Here are the code snippets and the error I get:
>

Then you were allowing it to be called automatically by PETSc, and it would
have been this time as well.


> //Setting up the section and linking it to DM:
> ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>             PetscInt point;
>             if(isPosInDomain(&testPoisson,i,j,0)) {
>                 ierr = DMDAGetCellPoint(dm, i, j, 0, &point);CHKERRQ(ierr);
>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
> No. of dofs associated with the point.
>             }
>         }
>     }
>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>     ierr = DMCreateMatrix(dm,&A);CHKERRQ(ierr);
>
>  //Set up KSP:
>     ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>     ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
>     ierr =
> KSPSetComputeOperators(ksp,computeMatrix2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>     ierr =
> KSPSetComputeRHS(ksp,computeRhs2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>     ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
>     ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);
>
> ------------------------------------------------------
> The computeMatrix2dSection function has:
>
> ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>     ierr = DMGetDefaultGlobalSection(da,&gs);CHKERRQ(ierr);
>     for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>         for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>             if (isPosInDomain(ctx,i,j)) {
>                 ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
>                 ierr = PetscSectionGetOffset(gs,point,&row);
>                 ierr = PetscSectionGetDof(gs,point,&dof);
>                 for(PetscInt cDof = 0; cDof < dof; ++cDof) {
>                     num = 0;
>                     row+=cDof;
>                     col[num] = row;         //(i,j) position
>                     v[num++] = -4;
>                     if(isPosInDomain(ctx,i+1,j)) {
>                         ierr =
> DMDAGetCellPoint(da,i+1,j,0,&point);CHKERRQ(ierr);
>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>                         col[num] += cDof;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i-1,j)) {
>                         ierr =
> DMDAGetCellPoint(da,i-1,j,0,&point);CHKERRQ(ierr);
>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>                         col[num] += cDof;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i,j+1)) {
>                         ierr =
> DMDAGetCellPoint(da,i,j+1,0,&point);CHKERRQ(ierr);
>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>                         col[num] += cDof;
>                         v[num++] = 1;
>                     }
>                     if(isPosInDomain(ctx,i,j-1)) {
>                         ierr =
> DMDAGetCellPoint(da,i,j-1,0,&point);CHKERRQ(ierr);
>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>                         col[num] += cDof;
>                         v[num++] = 1;
>                     }
>                     ierr =
> MatSetValues(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>                 }
>             }
>         }
>     }
>     ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>     ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>
> But this is not working. For e.g. for a 6X6 grid size I get the following
> error:
>

Okay, here is how to debug this:

  0) Go to a single scalar equations to make things easier

  1) Use MatView(A, PETSC_VIEWER_STDOUT_WORLD), which should output a 36
row matrix with
       the preallocated nonzero pattern, filled with zeros

  2) Make sure this is the pattern you want

  3) Run in the debugger with -start_in_debugger

  4) When you get the error, see

    a) If the (i, j) is one that should be setting a value

    b) Why this (i, j) was not preallocated

  Thanks,

     Matt


> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (0,6) caused a malloc!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development GIT revision:
> 61e5e40bb2c5bf2423e94b71a15fef47e411b0da  GIT Date: 2013-10-25 21:47:45
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: build/poissonIrregular on a arch-linux2-cxx-debug named
> edwards by bkhanal Thu Oct 31 19:23:58 2013
> [0]PETSC ERROR: Libraries linked from
> /home/bkhanal/Documents/softwares/petsc/arch-linux2-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Sat Oct 26 16:35:15 2013
> [0]PETSC ERROR: Configure options --download-mpich
> -download-f-blas-lapack=1 --download-metis --download-parmetis
> --download-superlu_dist --download-scalapack --download-mumps
> --download-hypre --with-clanguage=cxx
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 413 in
> src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: MatSetValues() line 1130 in src/mat/interface/matrix.c
> [0]PETSC ERROR: computeMatrix2dSection() line 318 in
> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
> [0]PETSC ERROR: KSPSetUp() line 228 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: main() line 598 in
> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>
>
>>    Matt
>>
>>
>>> However, now since I do not want the rows in the matrix corresponding to
>>> the points in the grid which do not lie in the computational domain. This
>>> means the size of the matrix will not necessarily equal the total number of
>>> cells in DMDA. So in the following code:
>>>
>>>  ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
>>>     ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>>             PetscInt point;
>>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>>> &point);CHKERRQ(ierr);
>>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof);
>>> // No. of dofs associated with the point.
>>>             }
>>>
>>>         }
>>>     }
>>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>>
>>> should I myself compute proper nC (i.e. total no. of points for which I
>>> want the matrix to have a corresponding row) before calling
>>> PetscSectionSetChart() or is the masking out of the points inside the loop
>>> with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
>>> And, when you write:
>>> PetscSectionGetOffset(gs, p, &ind);
>>>   row = ind < 0 ? -(ind+1) : ind;
>>>
>>> it seems you allow the possibility of getting a -ve ind when using
>>> PetscSectionGetOffset. When would an offset to a particular dof and point?
>>>
>>>
>>>
>>>
>>>>
>>>>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>>>                 num = 0;
>>>>>                 /*ierr =
>>>>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>>>>                 /*But I did now understand how I would emulate the
>>>>> row.c and col.c info with PetscSection*/
>>>>>                 row.i = i;  row.j = j;
>>>>>
>>>>
>>>> Here you would call
>>>>
>>>>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>>>>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>>>>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>>>>
>>>> This means that this cell has global rows = [row, row+dof).
>>>>
>>>>
>>>>>                  col[num].i = i; col[num].j = j;
>>>>>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator
>>>>> for only the values for only the points lying in the domain*/
>>>>>                     v[num++] = -4;
>>>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>>>                         col[num].i = i+1;   col[num].j = j;
>>>>>                         v[num++] = 1;
>>>>>                     }
>>>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>>>                         col[num].i = i-1;   col[num].j = j;
>>>>>                         v[num++] = 1;
>>>>>                     }
>>>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>>>                         col[num].i = i;     col[num].j = j+1;
>>>>>                         v[num++] = 1;
>>>>>                     }
>>>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>>>                         col[num].i = i;     col[num].j = j-1;
>>>>>                         v[num++] = 1;
>>>>>                     }
>>>>>                 } else {
>>>>>                     v[num++] = dScale;  /*set Dirichlet identity rows
>>>>> for points in the rectangle but outside the computational domain*/
>>>>>                 }
>>>>>
>>>>
>>>> You do the same thing for the columns, and then call
>>>>
>>>>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
>>>> INSERT_VALUES);CHKERRQ(ierr);
>>>>
>>>>
>>>>>                 ierr =
>>>>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>>>             }
>>>>>         }
>>>>>     }
>>>>>
>>>>>
>>>>>>     Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>    Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>      Matt
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might
>>>>>>>>>>>>>>> be the case. But is it always true that having identity rows in the system
>>>>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a
>>>>>>>>>>>>>> point. This is just decoupling these unknowns from the rest
>>>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/cb57b64d/attachment-0001.html>

From knepley at gmail.com  Thu Oct 31 14:01:39 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 14:01:39 -0500
Subject: [petsc-users] Fwd: MatZeroRowsColumns for mpibaij
In-Reply-To: <BDB09259-03B6-4C62-BA5C-373F20B0243E@anu.edu.au>
References: <5271BD65.1010507@imperial.ac.uk>
	<BDB09259-03B6-4C62-BA5C-373F20B0243E@anu.edu.au>
Message-ID: <CAMYG4GnAkhaqyLyJk4dwqwAh7APZSQUsikxxMBtfeMF6BRyr2w@mail.gmail.com>

On Wed, Oct 30, 2013 at 9:19 PM, Rhodri Davies <rhodri.davies at anu.edu.au>wrote:

>  Dear all,
>
>  Would you mind giving us an update on your thoughts re: this question?
> We're very keen to be able to use such functionally and it would be good to
> know if you're looking into this / have any suggestions?
>

I looked at the code. It does not look hard to make a MPIBAIJ version.
Since the scatter context for BAIJ is already exploded, most
of the MPIAIJ code goes straight through, it just a matter of handling the
blocking in a few places.

Here is what would be really helpful for us. Write a nice test. The code
would probably only take an hour of being careful. However,
the test assembly would be more. If you give us a nice test example for it,
writing it could happen pretty quickly.

  Thanks,

    Matt


> Best wishes,
>
>  Rhodri
>
>
> ---------------------------------------------------------------------------------------------------
> Research Fellow
> Research School of Earth Sciences
> The Australian National University
> Canberra, ACT, 0200, Australia
>
> T: +61 (0)2 6125 3643
> E: rhodri.davies at anu.edu.au
> W: http://rses.anu.edu.au/people/dr-rhodri-davies
>
> ---------------------------------------------------------------------------------------------------
>
>
>
> Begin forwarded message:
>
>  *From: *Stephan Kramer <s.kramer at imperial.ac.uk>
>  *Subject: **Fwd: [petsc-users] MatZeroRowsColumns for mpibaij*
>  *Date: *31 October 2013 13:16:05 GMT+11:00
>  *To: *Rhodri Davies <rhodri.davies at anu.edu.au>
>
>
>
>
> -------- Original Message --------
> Subject: [petsc-users] MatZeroRowsColumns for mpibaij
> Date: Wed, 19 Jun 2013 13:22:31 +0100
> From: Stephan Kramer <s.kramer at imperial.ac.uk>
> To: PETSc users list <petsc-users at mcs.anl.gov>
>
> Dear all,
>
> We have found the MatZeroRowsColumns() routine to be very useful for
> lifting boundary conditions. Unfortunately we also found that it isn't
> implemented for matrix type mpibaij and we're quite keen on
> keeping the baij structure as it seems to give us significant savings for
> DG problems (in particular the blocked sor). I had a quick look at the
> implementation of MatZeroRowsColums_MPIAIJ but couldn't
> really estimate how hard it would be to do something similar for mpibaij
> (an implementation for seqbaij exists already). Is this something that's
> likely to be looked at in the future?
>
> Stephan
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/3ad93a81/attachment.html>

From lu_qin_2000 at yahoo.com  Thu Oct 31 14:56:56 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 31 Oct 2013 12:56:56 -0700 (PDT)
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
Message-ID: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>

Hello,

I have successfully?installed a sequential PETSc's linear solver in a sequential program. The matrix has been created in CSR format?and it was passed to ?MatCreateSeqAIJWithArrays?(instead of using?MatSetValues). 

Now I want to use a parallel PETSc solver in the same sequential program, and use the same CRS matrix mentioned above. I intend to use MatCreateMPIAIJWithArrays, since I hope to pass the array direct to PETSc, instead of using MatSetValues. There a few questions:
?
1. Is there a sample code for using MatCreateMPIAIJWithArrays?
2. There seems to be a subroutine MatDistribute_MPIAIJ, but it is not in the PETSc manual. Where can I get the instruction how to use it? It would be nice if there is a sample code available.
3. Manual says the matrix will be copied by MatCreateMPIAIJWithArrays into internal format (while it is not when using MatCreateSeqAIJWithArrays), does this mean I have to call MatMPIAIJSetPreallocation first?
4. In sample code ex2.c in the manual, both MatMPIAIJSetPreallocation and MatSeqAIJSetPreallocation are called to preallocate the matrix A. Is not the first subroutine sufficient???
?
Many thanks for your help.
?
Regards,
Qin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/f07d3757/attachment.html>

From knepley at gmail.com  Thu Oct 31 15:01:40 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 15:01:40 -0500
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
Message-ID: <CAMYG4GmneT72wfUbTgiOzROFJAjcsB8jQ_RPLkYUu5vbfiU97w@mail.gmail.com>

On Thu, Oct 31, 2013 at 2:56 PM, Qin Lu <lu_qin_2000 at yahoo.com> wrote:

> Hello,
>
> I have successfully installed a sequential PETSc's linear solver in a
> sequential program. The matrix has been created in CSR format and it was
> passed to  MatCreateSeqAIJWithArrays (instead of using MatSetValues).
>
> Now I want to use a parallel PETSc solver in the same sequential program,
> and use the same CRS matrix mentioned above. I intend to use
> MatCreateMPIAIJWithArrays, since I hope to pass the array direct to PETSc,
> instead of using MatSetValues. There a few questions:
>

There is no preallocation when you do this. However, why do you want to do
this? It is generally a bad idea unless you
have a huge installed codebase that depends on that format.

  Matt


> 1. Is there a sample code for using MatCreateMPIAIJWithArrays?
> 2. There seems to be a subroutine MatDistribute_MPIAIJ, but it is not in
> the PETSc manual. Where can I get the instruction how to use it? It would
> be nice if there is a sample code available.
> 3. Manual says the matrix will be copied by MatCreateMPIAIJWithArrays into
> internal format (while it is not when using MatCreateSeqAIJWithArrays),
> does this mean I have to call MatMPIAIJSetPreallocation first?
> 4. In sample code ex2.c in the manual, both MatMPIAIJSetPreallocation and
> MatSeqAIJSetPreallocation are called to preallocate the matrix A. Is not
> the first subroutine sufficient?
>
> Many thanks for your help.
>
> Regards,
> Qin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/0ec285e6/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 31 15:06:55 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 14:06:55 -0600
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
Message-ID: <87sivhi4rk.fsf@mcs.anl.gov>

Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Hello,
>
> I have successfully?installed a sequential PETSc's linear solver in a sequential program. The matrix has been created in CSR format?and it was passed to ?MatCreateSeqAIJWithArrays?(instead of using?MatSetValues). 
>
> Now I want to use a parallel PETSc solver in the same sequential program, and use the same CRS matrix mentioned above. I intend to use MatCreateMPIAIJWithArrays, since I hope to pass the array direct to PETSc, instead of using MatSetValues. There a few questions:

Use MatSetValues.  It uses less memory and is more flexible.

> 1. Is there a sample code for using MatCreateMPIAIJWithArrays?

Please don't use this function.  It only exists to support legacy users
that were already using it.  The implementation calls MatSetValues
because the input format is not suitable to actually compute with.

> 2. There seems to be a subroutine MatDistribute_MPIAIJ, but it is not
> in the PETSc manual. Where can I get the instruction how to use it? It
> would be nice if there is a sample code available.

That is a private function.  Use MatSetValues.

> 3. Manual says the matrix will be copied by MatCreateMPIAIJWithArrays
> into internal format (while it is not when using
> MatCreateSeqAIJWithArrays), does this mean I have to call
> MatMPIAIJSetPreallocation first?

Not if you call MatCreateMPIAIJWithArrays, but to avoid using the extra
memory (and to simplify your code), you should compute preallocation and
then call MatSetValues once per row, per element, or whatever.

> 4. In sample code ex2.c in the manual, both MatMPIAIJSetPreallocation
> and MatSeqAIJSetPreallocation are called to preallocate the matrix
> A. Is not the first subroutine sufficient?

The example can run in serial or parallel.  Only the matching
preallocation is used, any others will be ignored.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/a138d00f/attachment.pgp>

From lu_qin_2000 at yahoo.com  Thu Oct 31 15:46:07 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 31 Oct 2013 13:46:07 -0700 (PDT)
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <87sivhi4rk.fsf@mcs.anl.gov>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
Message-ID: <1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>

Jed,
?
Thanks a lot for your nice suggestions. The CRS matrix has already been created by the program and I don't want to change that, do you mean I should read the arrays (i, j, a) and set coefficients row?by row using?MatSetValues? Will it be much slower than passing the arrays directly to MatCreateSeqAIJWithArrays or MatCreateMPIAIJWithArrays, especially when the matrix is big?
?
Thanks,
Qin



On Thursday, October 31, 2013 3:07 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
  
Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Hello,
>
> I have successfully?installed a sequential PETSc's linear solver in a sequential program. The matrix has been created in CSR format?and it was passed to ?MatCreateSeqAIJWithArrays?(instead of using?MatSetValues). 
>
> Now I want to use a parallel PETSc solver in the same sequential program, and use the same CRS matrix mentioned above. I intend to use MatCreateMPIAIJWithArrays, since I hope to pass the array direct to PETSc, instead of using MatSetValues. There a few questions:

Use MatSetValues.? It uses less memory and is more flexible.

> 1. Is there a sample code for using MatCreateMPIAIJWithArrays?

Please don't use this function.? It only exists to support legacy users
that were already using it.? The implementation calls MatSetValues
because the input format is not suitable to actually compute with.

> 2. There seems to be a subroutine MatDistribute_MPIAIJ, but it is not
> in the PETSc manual. Where can I get the instruction how to use it? It
> would be nice if there is a sample code available.

That is a private function.? Use MatSetValues.

> 3. Manual says the matrix will be copied by MatCreateMPIAIJWithArrays
> into internal format (while it is not when using
> MatCreateSeqAIJWithArrays), does this mean I have to call
> MatMPIAIJSetPreallocation first?

Not if you call MatCreateMPIAIJWithArrays, but to avoid using the extra
memory (and to simplify your code), you should compute preallocation and
then call MatSetValues once per row, per element, or whatever.


> 4. In sample code ex2.c in the manual, both MatMPIAIJSetPreallocation
> and MatSeqAIJSetPreallocation are called to preallocate the matrix
> A. Is not the first subroutine sufficient?

The example can run in serial or parallel.? Only the matching
preallocation is used, any others will be ignored.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/8cfb514f/attachment-0001.html>

From tsariysk at craft-tech.com  Thu Oct 31 15:47:45 2013
From: tsariysk at craft-tech.com (Ted Sariyski)
Date: Thu, 31 Oct 2013 16:47:45 -0400
Subject: [petsc-users] How to pass data to PETSc
Message-ID: <5272C1F1.2030008@craft-tech.com>

Hi,
I am about to use PETSc linear algebra. Two questions.

Q1: I have the stiffness matrix and the rhs vector assembled (outside) 
and in PETSc order. Do I still need to create a scatter:"ierr = 
VecScatterCreate(bb,isglobal,localX,islocal,&scatter)"?

Q2: I have PETSc vectors and matrix set:

ierr = VecCreate(MPI_COMM_WORLD,&bb);CHKERRQ(ierr);
ierr = VecSetFromOptions(bb);CHKERRQ(ierr);CHKPT
ierr = VecSetSizes(bb,Nvlocal,Nvglobal);CHKERRQ(ierr);
ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
ierr = MatSetSizes(A,Nvlocal,Nvlocal,Nvglobal,Nvglobal);CHKERRQ(ierr);

I passed the rhs myData to bb using VecSetValues:
ierr = VecSetValues(bb,Nvlocal,vertices,myData,INSERT_VALUES);CHKERRQ(ierr);

The matrix is in HYPRE sparse format. What is the recommended way to 
pass it to A?

Thanks in advance,
--Ted
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/c17ab25e/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 31 15:49:55 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 14:49:55 -0600
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
	<1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
Message-ID: <87ppqli2rw.fsf@mcs.anl.gov>

Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Jed, ? Thanks a lot for your nice suggestions. The CRS matrix has
> already been created by the program and I don't want to change that,
> do you mean I should read the arrays (i, j, a) and set coefficients
> row?by row using?MatSetValues? Will it be much slower than passing the
> arrays directly to MatCreateSeqAIJWithArrays or
> MatCreateMPIAIJWithArrays, especially when the matrix is big?

Are your *parallel* matrices already assembled in that form (which is
not suitable to compute with)?

In any case, MatCreateMPIAIJWithArrays is *implemented* by looping over
the rows calling MatSetValues.  The best is to generate the matrix by
row or element and insert at that time, but that is mostly for memory
reasons.  Copying the entries by row is not that expensive.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/47937d3c/attachment.pgp>

From jedbrown at mcs.anl.gov  Thu Oct 31 15:53:29 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 14:53:29 -0600
Subject: [petsc-users] How to pass data to PETSc
In-Reply-To: <5272C1F1.2030008@craft-tech.com>
References: <5272C1F1.2030008@craft-tech.com>
Message-ID: <87mwlpi2ly.fsf@mcs.anl.gov>

Ted Sariyski <tsariysk at craft-tech.com> writes:

> Hi,
> I am about to use PETSc linear algebra. Two questions.
>
> Q1: I have the stiffness matrix and the rhs vector assembled (outside) 
> and in PETSc order. Do I still need to create a scatter:"ierr = 
> VecScatterCreate(bb,isglobal,localX,islocal,&scatter)"?

What do you want to do with the Vec?

> Q2: I have PETSc vectors and matrix set:
>
> ierr = VecCreate(MPI_COMM_WORLD,&bb);CHKERRQ(ierr);
> ierr = VecSetFromOptions(bb);CHKERRQ(ierr);CHKPT
> ierr = VecSetSizes(bb,Nvlocal,Nvglobal);CHKERRQ(ierr);
> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> ierr = MatSetSizes(A,Nvlocal,Nvlocal,Nvglobal,Nvglobal);CHKERRQ(ierr);
>
> I passed the rhs myData to bb using VecSetValues:
> ierr = VecSetValues(bb,Nvlocal,vertices,myData,INSERT_VALUES);CHKERRQ(ierr);
>
> The matrix is in HYPRE sparse format. What is the recommended way to 
> pass it to A?

Best is to use MatSetValues while you are generating the matrix entries
(so that you don't store the Hypre format).  You can always do this
afterward.  (The Hypre matrix can't be used directly because Hypre does
not expose a complete set of functionality.  There is not a conversion
routine in PETSc, though it would be easy to add.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/a480e29f/attachment.pgp>

From lu_qin_2000 at yahoo.com  Thu Oct 31 15:56:22 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 31 Oct 2013 13:56:22 -0700 (PDT)
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <87ppqli2rw.fsf@mcs.anl.gov>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
	<1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
	<87ppqli2rw.fsf@mcs.anl.gov>
Message-ID: <1383252982.48442.YahooMailNeo@web160206.mail.bf1.yahoo.com>

Jed,
?
What about MatCreateSeqAIJWithArrays? Is it also implemented by looping over
the rows calling MatSetValues?
?
My CRS matrix is constructed in the master processor when using a parallel solver. Do I have to manually partition it (using metis, for instance) and distribute it to all processors using MPI, or PETSc has any subroutines to do this job?
?
Thanks,
Qin



On Thursday, October 31, 2013 3:50 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
  
Qin Lu <lu_qin_2000 at yahoo.com> writes:


> Jed, ? Thanks a lot for your nice suggestions. The CRS matrix has
> already been created by the program and I don't want to change that,
> do you mean I should read the arrays (i, j, a) and set coefficients
> row?by row using?MatSetValues? Will it be much slower than passing the
> arrays directly to MatCreateSeqAIJWithArrays or
> MatCreateMPIAIJWithArrays, especially when the matrix is big?

Are your *parallel* matrices already assembled in that form (which is
not suitable to compute with)?

In any case, MatCreateMPIAIJWithArrays is *implemented* by looping over
the rows calling MatSetValues.? The best is to generate the matrix by
row or element and insert at that time, but that is mostly for memory
reasons.? Copying the entries by row is not that expensive.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/c0ba1eeb/attachment.html>

From jedbrown at mcs.anl.gov  Thu Oct 31 16:07:41 2013
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 Oct 2013 15:07:41 -0600
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <1383252982.48442.YahooMailNeo@web160206.mail.bf1.yahoo.com>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
	<1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
	<87ppqli2rw.fsf@mcs.anl.gov>
	<1383252982.48442.YahooMailNeo@web160206.mail.bf1.yahoo.com>
Message-ID: <87habxi1ya.fsf@mcs.anl.gov>

Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Jed,
> ?
> What about MatCreateSeqAIJWithArrays? Is it also implemented by looping over
> the rows calling MatSetValues?

No, it uses the arrays directly because that format makes sense in
serial.

> My CRS matrix is constructed in the master processor when using a
> parallel solver. 

This will absolutely cripple your scalability.  If you're just trying to
use a few cores, then fine, but if you care about scalability, you need
to rethink your design.

> Do I have to manually partition it (using metis, for instance) and
> distribute it to all processors using MPI, or PETSc has any
> subroutines to do this job?

You should really partition the *mesh*, then assemble the local parts in
parallel.  The alternative you can use is to partition the matrix
entries (renumbering from your "native" ordering) and then call
MatSetValues (mapping the indices) from rank 0.  This part is not
scalable and may only make sense if you have a difficult problem or many
systems to solve.  Better to do it right and assemble in parallel.  It's
not difficult.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/ba63a702/attachment.pgp>

From lu_qin_2000 at yahoo.com  Thu Oct 31 16:44:09 2013
From: lu_qin_2000 at yahoo.com (Qin Lu)
Date: Thu, 31 Oct 2013 14:44:09 -0700 (PDT)
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <87habxi1ya.fsf@mcs.anl.gov>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
	<1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
	<87ppqli2rw.fsf@mcs.anl.gov>
	<1383252982.48442.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<87habxi1ya.fsf@mcs.anl.gov>
Message-ID: <1383255849.90291.YahooMailNeo@web160201.mail.bf1.yahoo.com>

Jed,

The program is sequential so the CSR matrix is only constructed in the master processor.?You said: "You should really partition the *mesh*, then assemble the local parts in parallel", do you mean I should parallelize the whole program?(using domain decomposition, for instance)? That will be my next step. But for now I need to make the parallel solver work in a sequential program.

In the context of partitioning/distributing the matrix, the manual says the first n0 number of rows should be on the first process, the second n1 rows should be on the second process, and so on. But when calling MatSetValues, each coefficient should use?GLOBAL (instead of local) row/column index (as showed in sample code ex2.c), right?

Thanks,
Qin



On Thursday, October 31, 2013 4:07 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

Qin Lu <lu_qin_2000 at yahoo.com> writes:

> Jed,
> ?
> What about MatCreateSeqAIJWithArrays? Is it also implemented by looping over
> the rows calling MatSetValues?

No, it uses the arrays directly because that format makes sense in
serial.

> My CRS matrix is constructed in the master processor when using a
> parallel solver. 

This will absolutely cripple your scalability.? If you're just trying to
use a few cores, then fine, but if you care about scalability, you need
to rethink your design.


> Do I have to manually partition it (using metis, for instance) and
> distribute it to all processors using MPI, or PETSc has any
> subroutines to do this job?

You should really partition the *mesh*, then assemble the local parts in
parallel.? The alternative you can use is to partition the matrix
entries (renumbering from your "native" ordering) and then call
MatSetValues (mapping the indices) from rank 0.? This part is not
scalable and may only make sense if you have a difficult problem or many
systems to solve.? Better to do it right and assemble in parallel.? It's
not difficult.?????

From bisheshkh at gmail.com  Thu Oct 31 17:02:11 2013
From: bisheshkh at gmail.com (Bishesh Khanal)
Date: Thu, 31 Oct 2013 23:02:11 +0100
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAMYG4GkDQLCXmWEcDJKZNSUnHbCpUicr0tr5cqaUxmuUYYWyVA@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
	<CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
	<CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>
	<CAEhex8i1iw+Nh7J3ECW+MTurwb-uWtw0hM_d-UnDxB8-H6LBJQ@mail.gmail.com>
	<CAMYG4GkDQLCXmWEcDJKZNSUnHbCpUicr0tr5cqaUxmuUYYWyVA@mail.gmail.com>
Message-ID: <CAEhex8jFmpZnAgjmBSR4brDaa2KVzd3J3hLpCgx1ZZgHQ-mxPw@mail.gmail.com>

On Thu, Oct 31, 2013 at 7:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 31, 2013 at 1:25 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>
>> I did not call DMCreateMatrix() before when using just dmda (and it is
>> working). In any case, I added a call to DMCreateMatrix() now but still
>> there are problems. Here are the code snippets and the error I get:
>>
>
> Then you were allowing it to be called automatically by PETSc, and it
> would have been this time as well.
>
>
>> //Setting up the section and linking it to DM:
>> ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>             PetscInt point;
>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>> &point);CHKERRQ(ierr);
>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof); //
>> No. of dofs associated with the point.
>>             }
>>         }
>>     }
>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>     ierr = DMCreateMatrix(dm,&A);CHKERRQ(ierr);
>>
>>  //Set up KSP:
>>     ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>     ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
>>     ierr =
>> KSPSetComputeOperators(ksp,computeMatrix2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>>     ierr =
>> KSPSetComputeRHS(ksp,computeRhs2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>>     ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>     ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);
>>
>> ------------------------------------------------------
>> The computeMatrix2dSection function has:
>>
>> ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>     ierr = DMGetDefaultGlobalSection(da,&gs);CHKERRQ(ierr);
>>     for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>         for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>             if (isPosInDomain(ctx,i,j)) {
>>                 ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
>>                 ierr = PetscSectionGetOffset(gs,point,&row);
>>                 ierr = PetscSectionGetDof(gs,point,&dof);
>>                 for(PetscInt cDof = 0; cDof < dof; ++cDof) {
>>                     num = 0;
>>                     row+=cDof;
>>                     col[num] = row;         //(i,j) position
>>                     v[num++] = -4;
>>                     if(isPosInDomain(ctx,i+1,j)) {
>>                         ierr =
>> DMDAGetCellPoint(da,i+1,j,0,&point);CHKERRQ(ierr);
>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>                         col[num] += cDof;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i-1,j)) {
>>                         ierr =
>> DMDAGetCellPoint(da,i-1,j,0,&point);CHKERRQ(ierr);
>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>                         col[num] += cDof;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i,j+1)) {
>>                         ierr =
>> DMDAGetCellPoint(da,i,j+1,0,&point);CHKERRQ(ierr);
>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>                         col[num] += cDof;
>>                         v[num++] = 1;
>>                     }
>>                     if(isPosInDomain(ctx,i,j-1)) {
>>                         ierr =
>> DMDAGetCellPoint(da,i,j-1,0,&point);CHKERRQ(ierr);
>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>                         col[num] += cDof;
>>                         v[num++] = 1;
>>                     }
>>                     ierr =
>> MatSetValues(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>                 }
>>             }
>>         }
>>     }
>>     ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>     ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>
>> But this is not working. For e.g. for a 6X6 grid size I get the following
>> error:
>>
>
> Okay, here is how to debug this:
>
>   0) Go to a single scalar equations to make things easier
>
>   1) Use MatView(A, PETSC_VIEWER_STDOUT_WORLD), which should output a 36
> row matrix with
>        the preallocated nonzero pattern, filled with zeros
>
>   2) Make sure this is the pattern you want
>
>   3) Run in the debugger with -start_in_debugger
>
>   4) When you get the error, see
>
>     a) If the (i, j) is one that should be setting a value
>
>     b) Why this (i, j) was not preallocated
>

Up to 4 (a), it is correct. There is a problem in the way DMDAGetCellPoint
and PetscSectionGetOffset works in this case scenario. I will try to
explain it below for the case of 4X4 grid.
First case:
If I set the computational domain to be all the points of the dmda grid,
(i.e. isPosInDomain(..,i,j,..) returns true for all i and j in the dmda
grid), the program runs fine and does not give any error.

Second case:
I want the computational domain to be some part of the whole grid. There is
a problem in this case.
The following test is in a case where,
isPosInDomain(..,i,j,..) returns true ONLY for (i,j) pairs of (2,1) (1,2)
(2,2) (3,2) and (3,2). The grid with its corresponding point number in the
petscsection is shown below:

12  13  14  15
8    9     10  11
4    5     6    7
0    1     2    3

of which 6, 9, 10, 11 and 14 correspond to the (i,j) points that returns
true for isPosInDomain(..,i,j,..)
MatView gives me the 16-row matrix with the star stencil non-zero structure
as expected.
The error I get is: new non-zero at (0,2) caused a malloc.

This error is for the value of (i,j) = (2,1), i.e. point 6.
The loop to set values in the matrix starts as:
for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
        for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
            if (isPosInDomain(ctx,i,j)) {
                ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
                ierr = PetscSectionGetOffset(gs,point,&row);

Now, what I wanted from the beginning is to create the matrix containing
the rows corresponding to only those points (i,j) which has true values for
isPosInDomain(ctx,i,j), that means in this case 5 row matrix. In the
current test, the first (i,j) that reaches DMDAGetCellPoint is (2,1), which
gives sets point variable to 6.
The line PetscSectionGetOffset(gs,point,&row) sets the value of row to 0.
So I think this where the inconsistency lies.
MatSetValues(A,1,&row,num,col,v,INSERT_VALUES) expects row variable and
col[] array to correspond to (i,j) based on DMDA, but PetsSectionGetOffset
gives result based on how we masked away some of the points from dmda grid.
Or am I missing something very basic here ?




>   Thanks,
>
>      Matt
>
>
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: New nonzero at (0,6) caused a malloc!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Development GIT revision:
>> 61e5e40bb2c5bf2423e94b71a15fef47e411b0da  GIT Date: 2013-10-25 21:47:45
>> -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: build/poissonIrregular on a arch-linux2-cxx-debug named
>> edwards by bkhanal Thu Oct 31 19:23:58 2013
>> [0]PETSC ERROR: Libraries linked from
>> /home/bkhanal/Documents/softwares/petsc/arch-linux2-cxx-debug/lib
>> [0]PETSC ERROR: Configure run at Sat Oct 26 16:35:15 2013
>> [0]PETSC ERROR: Configure options --download-mpich
>> -download-f-blas-lapack=1 --download-metis --download-parmetis
>> --download-superlu_dist --download-scalapack --download-mumps
>> --download-hypre --with-clanguage=cxx
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 413 in
>> src/mat/impls/aij/seq/aij.c
>> [0]PETSC ERROR: MatSetValues() line 1130 in src/mat/interface/matrix.c
>> [0]PETSC ERROR: computeMatrix2dSection() line 318 in
>> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
>> [0]PETSC ERROR: KSPSetUp() line 228 in src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: main() line 598 in
>> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
>> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>> [unset]: aborting job:
>> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>>
>>
>>>    Matt
>>>
>>>
>>>> However, now since I do not want the rows in the matrix corresponding
>>>> to the points in the grid which do not lie in the computational domain.
>>>> This means the size of the matrix will not necessarily equal the total
>>>> number of cells in DMDA. So in the following code:
>>>>
>>>>  ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
>>>>     ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>>>             PetscInt point;
>>>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>>>> &point);CHKERRQ(ierr);
>>>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof);
>>>> // No. of dofs associated with the point.
>>>>             }
>>>>
>>>>         }
>>>>     }
>>>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>>>
>>>> should I myself compute proper nC (i.e. total no. of points for which I
>>>> want the matrix to have a corresponding row) before calling
>>>> PetscSectionSetChart() or is the masking out of the points inside the loop
>>>> with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
>>>> And, when you write:
>>>> PetscSectionGetOffset(gs, p, &ind);
>>>>   row = ind < 0 ? -(ind+1) : ind;
>>>>
>>>> it seems you allow the possibility of getting a -ve ind when using
>>>> PetscSectionGetOffset. When would an offset to a particular dof and point?
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>>>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>>>>                 num = 0;
>>>>>>                 /*ierr =
>>>>>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>>>>>                 /*But I did now understand how I would emulate the
>>>>>> row.c and col.c info with PetscSection*/
>>>>>>                 row.i = i;  row.j = j;
>>>>>>
>>>>>
>>>>> Here you would call
>>>>>
>>>>>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>>>>>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>>>>>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>>>>>
>>>>> This means that this cell has global rows = [row, row+dof).
>>>>>
>>>>>
>>>>>>                  col[num].i = i; col[num].j = j;
>>>>>>                 if (isPosInDomain(ctx,i,j)) {      /*put the operator
>>>>>> for only the values for only the points lying in the domain*/
>>>>>>                     v[num++] = -4;
>>>>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>>>>                         col[num].i = i+1;   col[num].j = j;
>>>>>>                         v[num++] = 1;
>>>>>>                     }
>>>>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>>>>                         col[num].i = i-1;   col[num].j = j;
>>>>>>                         v[num++] = 1;
>>>>>>                     }
>>>>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>>>>                         col[num].i = i;     col[num].j = j+1;
>>>>>>                         v[num++] = 1;
>>>>>>                     }
>>>>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>>>>                         col[num].i = i;     col[num].j = j-1;
>>>>>>                         v[num++] = 1;
>>>>>>                     }
>>>>>>                 } else {
>>>>>>                     v[num++] = dScale;  /*set Dirichlet identity rows
>>>>>> for points in the rectangle but outside the computational domain*/
>>>>>>                 }
>>>>>>
>>>>>
>>>>> You do the same thing for the columns, and then call
>>>>>
>>>>>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
>>>>> INSERT_VALUES);CHKERRQ(ierr);
>>>>>
>>>>>
>>>>>>                 ierr =
>>>>>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>>>>             }
>>>>>>         }
>>>>>>     }
>>>>>>
>>>>>>
>>>>>>>     Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>    Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>    Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>>>>>> the rows you do not use on the full cube. It will not hurt
>>>>>>>>>>>>>>>>> scalability or convergence.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might
>>>>>>>>>>>>>>>> be the case. But is it always true that having identity rows in the system
>>>>>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a
>>>>>>>>>>>>>>> point. This is just decoupling these unknowns from the rest
>>>>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing more
>>>>>>>>>>>>>>>> experiments to confirm) while solving my staggered-grid 3D stokes flow with
>>>>>>>>>>>>>>>> schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Matt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  Anyway please provide me with some pointers so that I can
>>>>>>>>>>>>>>>> start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/5164a25c/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Oct 31 17:06:04 2013
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 31 Oct 2013 17:06:04 -0500
Subject: [petsc-users] Use parallel PETSc linear solver in a sequential
	program
In-Reply-To: <1383255849.90291.YahooMailNeo@web160201.mail.bf1.yahoo.com>
References: <1383249416.14014.YahooMailNeo@web160204.mail.bf1.yahoo.com>
	<87sivhi4rk.fsf@mcs.anl.gov>
	<1383252367.97431.YahooMailNeo@web160202.mail.bf1.yahoo.com>
	<87ppqli2rw.fsf@mcs.anl.gov>
	<1383252982.48442.YahooMailNeo@web160206.mail.bf1.yahoo.com>
	<87habxi1ya.fsf@mcs.anl.gov>
	<1383255849.90291.YahooMailNeo@web160201.mail.bf1.yahoo.com>
Message-ID: <0D90C748-57FC-44A0-B78E-67474555A155@mcs.anl.gov>


On Oct 31, 2013, at 4:44 PM, Qin Lu <lu_qin_2000 at yahoo.com> wrote:

> Jed,
> 
> The program is sequential so the CSR matrix is only constructed in the master processor. You said: "You should really partition the *mesh*, then assemble the local parts in parallel", do you mean I should parallelize the whole program (using domain decomposition, for instance)? That will be my next step. But for now I need to make the parallel solver work in a sequential program.

   This is really not something that the MPI parallel programming model is in anyway suited towards. The process of taking your sequential code and its sequential matrix and making the matrix be parallel is completely wasted effort instead 

   if you want to ?try out? parallel solvers to see how they scale on your matrices then you should 

1)   Run your sequential program and have it use MatView() and VecView() on the matrix (ices) and vector(s) right hand side with the binary viewer into a file

2)   Use src/ksp/ksp/examples/tutorials/ex10.c to load the matrix and vector from the file and solve the system in parallel on any number of processes.

   
   If you are happy with how the parallel linear solvers work then you rewrite the code to handle the mesh in parallel and generate the matrix in parallel using MatSetValues() etc.

   Barry

> 
> In the context of partitioning/distributing the matrix, the manual says the first n0 number of rows should be on the first process, the second n1 rows should be on the second process, and so on. But when calling MatSetValues, each coefficient should use GLOBAL (instead of local) row/column index (as showed in sample code ex2.c), right?
> 
> Thanks,
> Qin
> 
> 
> 
> On Thursday, October 31, 2013 4:07 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> 
> Qin Lu <lu_qin_2000 at yahoo.com> writes:
> 
>> Jed,
>>  
>> What about MatCreateSeqAIJWithArrays? Is it also implemented by looping over
>> the rows calling MatSetValues?
> 
> No, it uses the arrays directly because that format makes sense in
> serial.
> 
>> My CRS matrix is constructed in the master processor when using a
>> parallel solver. 
> 
> This will absolutely cripple your scalability.  If you're just trying to
> use a few cores, then fine, but if you care about scalability, you need
> to rethink your design.
> 
> 
>> Do I have to manually partition it (using metis, for instance) and
>> distribute it to all processors using MPI, or PETSc has any
>> subroutines to do this job?
> 
> You should really partition the *mesh*, then assemble the local parts in
> parallel.  The alternative you can use is to partition the matrix
> entries (renumbering from your "native" ordering) and then call
> MatSetValues (mapping the indices) from rank 0.  This part is not
> scalable and may only make sense if you have a difficult problem or many
> systems to solve.  Better to do it right and assemble in parallel.  It's
> not difficult.     


From knepley at gmail.com  Thu Oct 31 17:34:13 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 17:34:13 -0500
Subject: [petsc-users] using petsc tools to solve isolated irregular
 domains with finite difference
In-Reply-To: <CAEhex8jFmpZnAgjmBSR4brDaa2KVzd3J3hLpCgx1ZZgHQ-mxPw@mail.gmail.com>
References: <CAEhex8gA==Fg8P6g0WD8TOrqi7wKTFt=qL70VTt=5eopTQHjLw@mail.gmail.com>
	<CAMYG4G=4e3CGK=xcMZ7o3KSMg80oP1w0i9hm3xw1fzSVohirEw@mail.gmail.com>
	<CAEhex8hHcCz7bgnKPctskNShtcgdtP-WDpXB7ndGXnzfQJhMeg@mail.gmail.com>
	<CAMYG4Gm443X6g14+zLnoH=U7jZh2BEmJgj0uBj77EZ2+duweyg@mail.gmail.com>
	<CAEhex8ijWi=P2qRDKQQG21vjPDOss3e5Q0KVzQJy7N3Kn9Q8NA@mail.gmail.com>
	<CAMYG4GnDeJ8_=mJvukY92qJVO=pzZKcL5uKbY+ANUNdqhaNzUQ@mail.gmail.com>
	<CAEhex8i1NGR9KGgAfEA=fi0qBJO7Dwdg=4J0M+PKB5E=0nNkWA@mail.gmail.com>
	<CAMYG4GmYPttJLCUhd3VGcB7KbRrXq-ScycWcgvtd6MKMOujYTw@mail.gmail.com>
	<CAEhex8iZcNQBUkXT8AhYWOgkhFAAHpN6rMy0s+vOSjLSHoChgQ@mail.gmail.com>
	<CAMYG4GmPTRboVKNE5quWcf1xj3TPHB8OFh-8Td+H-N41dvgi1w@mail.gmail.com>
	<CAEhex8jwzmnr0J1qLUJu7LsBGezzvJsgOPAVq9N3h4ww4n=REw@mail.gmail.com>
	<CAMYG4GnAMj1zP89H+VsmVeCySd_xh2hT0RD2bBMUOn2aqMCJOw@mail.gmail.com>
	<CAEhex8jH_ie4_WPijJcGy7TvAUZBTwez3dC-wRfpoLyRVW3Row@mail.gmail.com>
	<CAMYG4Gkvfso0rwErYrd96Up36uuCe9-FNWmQ8ooFLpbWPYtKnA@mail.gmail.com>
	<CAEhex8ih4XXT5xSUs=D1iUHohKdTO4omH1LJ=GbOZqBcqPCgkg@mail.gmail.com>
	<CAMYG4Gkb-1zaU7__zHo9s_HTzWhHGwnwo-XEyTg1addw_h6uDA@mail.gmail.com>
	<CAEhex8i1iw+Nh7J3ECW+MTurwb-uWtw0hM_d-UnDxB8-H6LBJQ@mail.gmail.com>
	<CAMYG4GkDQLCXmWEcDJKZNSUnHbCpUicr0tr5cqaUxmuUYYWyVA@mail.gmail.com>
	<CAEhex8jFmpZnAgjmBSR4brDaa2KVzd3J3hLpCgx1ZZgHQ-mxPw@mail.gmail.com>
Message-ID: <CAMYG4GnRFL29vM-vcj1mw+dw4xTSuN1c--5s7NysPNY2aEoAFA@mail.gmail.com>

On Thu, Oct 31, 2013 at 5:02 PM, Bishesh Khanal <bisheshkh at gmail.com> wrote:

>
>
>
> On Thu, Oct 31, 2013 at 7:43 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Thu, Oct 31, 2013 at 1:25 PM, Bishesh Khanal <bisheshkh at gmail.com>wrote:
>>
>>> I did not call DMCreateMatrix() before when using just dmda (and it is
>>> working). In any case, I added a call to DMCreateMatrix() now but still
>>> there are problems. Here are the code snippets and the error I get:
>>>
>>
>> Then you were allowing it to be called automatically by PETSc, and it
>> would have been this time as well.
>>
>>
>>> //Setting up the section and linking it to DM:
>>> ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>>             PetscInt point;
>>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>>> &point);CHKERRQ(ierr);
>>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof);
>>> // No. of dofs associated with the point.
>>>             }
>>>         }
>>>     }
>>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>>     ierr = DMCreateMatrix(dm,&A);CHKERRQ(ierr);
>>>
>>>  //Set up KSP:
>>>     ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
>>>     ierr = KSPSetDM(ksp,dm);CHKERRQ(ierr);
>>>     ierr =
>>> KSPSetComputeOperators(ksp,computeMatrix2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>>>     ierr =
>>> KSPSetComputeRHS(ksp,computeRhs2dSection,(void*)&testPoisson);CHKERRQ(ierr);
>>>     ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);
>>>     ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);
>>>
>>> ------------------------------------------------------
>>> The computeMatrix2dSection function has:
>>>
>>> ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);
>>>     ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
>>>     ierr = DMGetDefaultGlobalSection(da,&gs);CHKERRQ(ierr);
>>>     for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>         for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>             if (isPosInDomain(ctx,i,j)) {
>>>                 ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
>>>                 ierr = PetscSectionGetOffset(gs,point,&row);
>>>                 ierr = PetscSectionGetDof(gs,point,&dof);
>>>                 for(PetscInt cDof = 0; cDof < dof; ++cDof) {
>>>                     num = 0;
>>>                     row+=cDof;
>>>                     col[num] = row;         //(i,j) position
>>>                     v[num++] = -4;
>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>                         ierr =
>>> DMDAGetCellPoint(da,i+1,j,0,&point);CHKERRQ(ierr);
>>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>>                         col[num] += cDof;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>                         ierr =
>>> DMDAGetCellPoint(da,i-1,j,0,&point);CHKERRQ(ierr);
>>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>>                         col[num] += cDof;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>                         ierr =
>>> DMDAGetCellPoint(da,i,j+1,0,&point);CHKERRQ(ierr);
>>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>>                         col[num] += cDof;
>>>                         v[num++] = 1;
>>>                     }
>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>                         ierr =
>>> DMDAGetCellPoint(da,i,j-1,0,&point);CHKERRQ(ierr);
>>>                         ierr = PetscSectionGetOffset(gs,point,&col[num]);
>>>                         col[num] += cDof;
>>>                         v[num++] = 1;
>>>                     }
>>>                     ierr =
>>> MatSetValues(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>                 }
>>>             }
>>>         }
>>>     }
>>>     ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>     ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>
>>> But this is not working. For e.g. for a 6X6 grid size I get the
>>> following error:
>>>
>>
>> Okay, here is how to debug this:
>>
>>   0) Go to a single scalar equations to make things easier
>>
>>   1) Use MatView(A, PETSC_VIEWER_STDOUT_WORLD), which should output a 36
>> row matrix with
>>        the preallocated nonzero pattern, filled with zeros
>>
>>   2) Make sure this is the pattern you want
>>
>>   3) Run in the debugger with -start_in_debugger
>>
>>   4) When you get the error, see
>>
>>     a) If the (i, j) is one that should be setting a value
>>
>>     b) Why this (i, j) was not preallocated
>>
>
> Up to 4 (a), it is correct. There is a problem in the way DMDAGetCellPoint
> and PetscSectionGetOffset works in this case scenario. I will try to
> explain it below for the case of 4X4 grid.
> First case:
> If I set the computational domain to be all the points of the dmda grid,
> (i.e. isPosInDomain(..,i,j,..) returns true for all i and j in the dmda
> grid), the program runs fine and does not give any error.
>
> Second case:
> I want the computational domain to be some part of the whole grid. There
> is a problem in this case.
> The following test is in a case where,
> isPosInDomain(..,i,j,..) returns true ONLY for (i,j) pairs of (2,1) (1,2)
> (2,2) (3,2) and (3,2). The grid with its corresponding point number in the
> petscsection is shown below:
>
> 12  13  14  15
> 8    9     10  11
> 4    5     6    7
> 0    1     2    3
>
> of which 6, 9, 10, 11 and 14 correspond to the (i,j) points that returns
> true for isPosInDomain(..,i,j,..)
> MatView gives me the 16-row matrix with the star stencil non-zero
> structure as expected.
> The error I get is: new non-zero at (0,2) caused a malloc.
>
> This error is for the value of (i,j) = (2,1), i.e. point 6.
> The loop to set values in the matrix starts as:
> for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>         for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>             if (isPosInDomain(ctx,i,j)) {
>                 ierr = DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);
>                 ierr = PetscSectionGetOffset(gs,point,&row);
>
> Now, what I wanted from the beginning is to create the matrix containing
> the rows corresponding to only those points (i,j) which has true values for
> isPosInDomain(ctx,i,j), that means in this case 5 row matrix. In the
> current test, the first (i,j) that reaches DMDAGetCellPoint is (2,1), which
> gives sets point variable to 6.
> The line PetscSectionGetOffset(gs,point,&row) sets the value of row to 0.
> So I think this where the inconsistency lies.
> MatSetValues(A,1,&row,num,col,v,INSERT_VALUES) expects row variable and
> col[] array to correspond to (i,j) based on DMDA, but PetsSectionGetOffset
> gives result based on how we masked away some of the points from dmda grid.
> Or am I missing something very basic here ?
>

You are missing something basic. The row is correctly identified as 0, and
that means the 2 is
point 10. The problem must be preallocation. First, you should send the
result of MatView().

   Matt


>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: New nonzero at (0,6) caused a malloc!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Development GIT revision:
>>> 61e5e40bb2c5bf2423e94b71a15fef47e411b0da  GIT Date: 2013-10-25 21:47:45
>>> -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: build/poissonIrregular on a arch-linux2-cxx-debug named
>>> edwards by bkhanal Thu Oct 31 19:23:58 2013
>>> [0]PETSC ERROR: Libraries linked from
>>> /home/bkhanal/Documents/softwares/petsc/arch-linux2-cxx-debug/lib
>>> [0]PETSC ERROR: Configure run at Sat Oct 26 16:35:15 2013
>>> [0]PETSC ERROR: Configure options --download-mpich
>>> -download-f-blas-lapack=1 --download-metis --download-parmetis
>>> --download-superlu_dist --download-scalapack --download-mumps
>>> --download-hypre --with-clanguage=cxx
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 413 in
>>> src/mat/impls/aij/seq/aij.c
>>> [0]PETSC ERROR: MatSetValues() line 1130 in src/mat/interface/matrix.c
>>> [0]PETSC ERROR: computeMatrix2dSection() line 318 in
>>> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
>>> [0]PETSC ERROR: KSPSetUp() line 228 in src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: KSPSolve() line 399 in src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: main() line 598 in
>>> /user/bkhanal/home/works/poissonIrregular/poissonIrregular.cxx
>>> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>>> [unset]: aborting job:
>>> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>>>
>>>
>>>>    Matt
>>>>
>>>>
>>>>> However, now since I do not want the rows in the matrix corresponding
>>>>> to the points in the grid which do not lie in the computational domain.
>>>>> This means the size of the matrix will not necessarily equal the total
>>>>> number of cells in DMDA. So in the following code:
>>>>>
>>>>>  ierr = DMDAGetLocalInfo(dm,&info);CHKERRQ(ierr);
>>>>>     ierr = PetscSectionCreate(PETSC_COMM_WORLD, &s);CHKERRQ(ierr);
>>>>>     ierr = DMDAGetNumCells(dm, NULL, NULL, NULL, &nC);CHKERRQ(ierr);
>>>>>     ierr = PetscSectionSetChart(s, 0, nC);CHKERRQ(ierr);
>>>>>     for (PetscInt j = info.ys; j < info.ys+info.ym; ++j) {
>>>>>         for (PetscInt i = info.xs; i < info.xs+info.xm; ++i) {
>>>>>             PetscInt point;
>>>>>             if(isPosInDomain(&testPoisson,i,j,0)) {
>>>>>                 ierr = DMDAGetCellPoint(dm, i, j, 0,
>>>>> &point);CHKERRQ(ierr);
>>>>>                 ierr = PetscSectionSetDof(s, point, testPoisson.mDof);
>>>>> // No. of dofs associated with the point.
>>>>>             }
>>>>>
>>>>>         }
>>>>>     }
>>>>>     ierr = PetscSectionSetUp(s);CHKERRQ(ierr);
>>>>>     ierr = DMSetDefaultSection(dm, s);CHKERRQ(ierr);
>>>>>     ierr = PetscSectionDestroy(&s);CHKERRQ(ierr);
>>>>>
>>>>> should I myself compute proper nC (i.e. total no. of points for which
>>>>> I want the matrix to have a corresponding row) before calling
>>>>> PetscSectionSetChart() or is the masking out of the points inside the loop
>>>>> with if(isPosInDomain(&testPoisson,i,j,0)) sufficient ?
>>>>> And, when you write:
>>>>> PetscSectionGetOffset(gs, p, &ind);
>>>>>   row = ind < 0 ? -(ind+1) : ind;
>>>>>
>>>>> it seems you allow the possibility of getting a -ve ind when using
>>>>> PetscSectionGetOffset. When would an offset to a particular dof and point?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>         for(PetscInt j = info.ys; j<info.ys+info.ym; ++j) {
>>>>>>>             for(PetscInt i = info.xs; i<info.xs+info.xm; ++i) {
>>>>>>>                 num = 0;
>>>>>>>                 /*ierr =
>>>>>>> DMDAGetCellPoint(da,i,j,0,&point);CHKERRQ(ierr);*/ /*Get the PetscPoint*/
>>>>>>>                 /*But I did now understand how I would emulate the
>>>>>>> row.c and col.c info with PetscSection*/
>>>>>>>                 row.i = i;  row.j = j;
>>>>>>>
>>>>>>
>>>>>> Here you would call
>>>>>>
>>>>>>    ierr = DMDAGetCellPoint(da, i, j, 0, &cell);CHKERRQ(ierr);
>>>>>>    ierr = PetscSectionGetOffset(da, cell, &row);CHKERRQ(ierr);
>>>>>>    ierr = PetscSectionGetDof(da, cell, &dof);CHKERRQ(ierr);
>>>>>>
>>>>>> This means that this cell has global rows = [row, row+dof).
>>>>>>
>>>>>>
>>>>>>>                  col[num].i = i; col[num].j = j;
>>>>>>>                 if (isPosInDomain(ctx,i,j)) {      /*put the
>>>>>>> operator for only the values for only the points lying in the domain*/
>>>>>>>                     v[num++] = -4;
>>>>>>>                     if(isPosInDomain(ctx,i+1,j)) {
>>>>>>>                         col[num].i = i+1;   col[num].j = j;
>>>>>>>                         v[num++] = 1;
>>>>>>>                     }
>>>>>>>                     if(isPosInDomain(ctx,i-1,j)) {
>>>>>>>                         col[num].i = i-1;   col[num].j = j;
>>>>>>>                         v[num++] = 1;
>>>>>>>                     }
>>>>>>>                     if(isPosInDomain(ctx,i,j+1)) {
>>>>>>>                         col[num].i = i;     col[num].j = j+1;
>>>>>>>                         v[num++] = 1;
>>>>>>>                     }
>>>>>>>                     if(isPosInDomain(ctx,i,j-1)) {
>>>>>>>                         col[num].i = i;     col[num].j = j-1;
>>>>>>>                         v[num++] = 1;
>>>>>>>                     }
>>>>>>>                 } else {
>>>>>>>                     v[num++] = dScale;  /*set Dirichlet identity
>>>>>>> rows for points in the rectangle but outside the computational domain*/
>>>>>>>                 }
>>>>>>>
>>>>>>
>>>>>> You do the same thing for the columns, and then call
>>>>>>
>>>>>>   ierr = MatSetValues(A, dof, rows, num*dof, cols, v,
>>>>>> INSERT_VALUES);CHKERRQ(ierr);
>>>>>>
>>>>>>
>>>>>>>                 ierr =
>>>>>>> MatSetValuesStencil(A,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>>>>>             }
>>>>>>>         }
>>>>>>>     }
>>>>>>>
>>>>>>>
>>>>>>>>     Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>    Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>    Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>   Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      Matt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  If not, just put the identity into
>>>>>>>>>>>>>>>>>> the rows you do not use on the full cube. It will not
>>>>>>>>>>>>>>>>>> hurt scalability or convergence.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In the case of Poisson with Dirichlet condition this might
>>>>>>>>>>>>>>>>> be the case. But is it always true that having identity rows in the system
>>>>>>>>>>>>>>>>> matrix will not hurt convergence ? I thought otherwise for the following
>>>>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>>> 1)  Having read Jed's answer here :
>>>>>>>>>>>>>>>>> http://scicomp.stackexchange.com/questions/3426/why-is-pinning-a-point-to-remove-a-null-space-bad/3427#3427
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jed is talking about a constraint on a the pressure at a
>>>>>>>>>>>>>>>> point. This is just decoupling these unknowns from the rest
>>>>>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) Some observation I am getting (but I am still doing
>>>>>>>>>>>>>>>>> more experiments to confirm) while solving my staggered-grid 3D stokes flow
>>>>>>>>>>>>>>>>> with schur complement and using -pc_type gamg for A00 matrix. Putting the
>>>>>>>>>>>>>>>>> identity rows for dirichlet boundaries and for ghost cells seemed to have
>>>>>>>>>>>>>>>>> effects on its convergence. I'm hoping once I know how to use PetscSection,
>>>>>>>>>>>>>>>>> I can get rid of using ghost cells method for the staggered grid and get
>>>>>>>>>>>>>>>>> rid of the identity rows too.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It can change the exact iteration, but it does not make the
>>>>>>>>>>>>>>>> matrix conditioning worse.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Matt
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  Anyway please provide me with some pointers so that I
>>>>>>>>>>>>>>>>> can start trying with petscsection on top of a dmda, in the beginning for
>>>>>>>>>>>>>>>>> non-staggered case.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>   Matt
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Bishesh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they
>>>>>>>>>>>>>>>>>> begin their experiments is infinitely more interesting than any results to
>>>>>>>>>>>>>>>>>> which their experiments lead.
>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/e4f37919/attachment-0001.html>

From danyang.su at gmail.com  Thu Oct 31 18:54:21 2013
From: danyang.su at gmail.com (Danyang Su)
Date: Thu, 31 Oct 2013 16:54:21 -0700
Subject: [petsc-users] Speedup problem when using OpenMP?
Message-ID: <5272EDAD.2070001@gmail.com>

Hi All,

I have a question on the speedup of PETSc when using OpenMP. I can get 
good speedup when using MPI, but no speedup when using OpenMP.
The example is ex2f with m=100 and n=100. The number of available 
processors is 16 (32 threads) and the OS is Windows Server 2012. The log 
files for 4 and 8 processors are attached.

The commands I used to run with 4 processors are as follows:
Run using MPI
mpiexec -n 4 Petsc-windows-ex2f.exe -m 100 -n 100 -log_summary 
log_100x100_mpi_p4.log

Run using OpenMP
Petsc-windows-ex2f.exe -threadcomm_type openmp -threadcomm_nthreads 4 -m 
100 -n 100 -log_summary log_100x100_openmp_p4.log

The PETSc used for this test is PETSc for Windows 
http://www.mic-tc.ch/downloads/PETScForWindows.zip, but I guess this is 
not the problem because the same problem exists when I use PETSc-dev in 
Cygwin. I don't know if this problem exists in Linux, would anybody help 
to test?

Thanks and regards,

Danyang
-------------- next part --------------
!
!  Description: Solves a linear system in parallel with KSP (Fortran code).
!               Also shows how to set a user-defined monitoring routine.
!
!
!/*T
!  Concepts: KSP^basic parallel example
!  Concepts: KSP^setting a user-defined monitoring routine
!  Processors: n
!T*/
!
! -----------------------------------------------------------------------

      program main
      implicit none

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                    Include files
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
!  This program uses CPP for preprocessing, as indicated by the use of
!  PETSc include files in the directory petsc/include/finclude.  This
!  convention enables use of the CPP preprocessor, which allows the use
!  of the #include statements that define PETSc objects and variables.
!
!  Use of the conventional Fortran include statements is also supported
!  In this case, the PETsc include files are located in the directory
!  petsc/include/foldinclude.
!
!  Since one must be very careful to include each file no more than once
!  in a Fortran routine, application programmers must exlicitly list
!  each file needed for the various PETSc components within their
!  program (unlike the C/C++ interface).
!
!  See the Fortran section of the PETSc users manual for details.
!
!  The following include statements are required for KSP Fortran programs:
!     petscsys.h       - base PETSc routines
!     petscvec.h    - vectors
!     petscmat.h    - matrices
!     petscpc.h     - preconditioners
!     petscksp.h    - Krylov subspace methods
!  Additional include statements may be needed if using additional
!  PETSc routines in a Fortran program, e.g.,
!     petscviewer.h - viewers
!     petscis.h     - index sets
!

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscpc.h>
#include <finclude/petscksp.h>

!
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                   Variable declarations
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!
!  Variables:
!     ksp     - linear solver context
!     ksp      - Krylov subspace method context
!     pc       - preconditioner context
!     x, b, u  - approx solution, right-hand-side, exact solution vectors
!     A        - matrix that defines linear system
!     its      - iterations for convergence
!     norm     - norm of error in solution
!     rctx     - random number generator context
!
!  Note that vectors are declared as PETSc "Vec" objects.  These vectors
!  are mathematical objects that contain more than just an array of
!  double precision numbers. I.e., vectors in PETSc are not just
!        double precision x(*).
!  However, local vector data can be easily accessed via VecGetArray().
!  See the Fortran section of the PETSc users manual for details.
!
      double precision  norm
      PetscInt  i,j,II,JJ,m,n,its
      PetscInt  Istart,Iend,ione
      PetscErrorCode ierr
      PetscMPIInt     rank,size
      PetscBool   flg
      PetscScalar v,one,neg_one
      Vec         x,b,u
      Mat         A
      KSP         ksp
      PetscRandom rctx

!  These variables are not currently used.
!      PC          pc
!      PCType      ptype
!      double precision tol


!  Note: Any user-defined Fortran routines (such as MyKSPMonitor)
!  MUST be declared as external.

      external MyKSPMonitor,MyKSPConverged

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                 Beginning of program
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      call PetscInitialize(Petsc_Null_Character,ierr)
      m = 3
      n = 3
      one  = 1.0
      neg_one = -1.0
      ione    = 1
      call PetscOptionsGetInt(Petsc_Null_Character,'-m',m,flg,ierr)
      call PetscOptionsGetInt(Petsc_Null_Character,'-n',n,flg,ierr)
      call MPI_Comm_rank(Petsc_Comm_World,rank,ierr)
      call MPI_Comm_size(Petsc_Comm_World,size,ierr)

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!      Compute the matrix and right-hand-side vector that define
!      the linear system, Ax = b.
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Create parallel matrix, specifying only its global dimensions.
!  When using MatCreate(), the matrix format can be specified at
!  runtime. Also, the parallel partitioning of the matrix is
!  determined by PETSc at runtime.

      call MatCreate(Petsc_Comm_World,A,ierr)
      call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n,ierr)
      call MatSetFromOptions(A,ierr)
      call MatSetUp(A,ierr)

!  Currently, all PETSc parallel matrix formats are partitioned by
!  contiguous chunks of rows across the processors.  Determine which
!  rows of the matrix are locally owned.

      call MatGetOwnershipRange(A,Istart,Iend,ierr)

!  Set matrix elements for the 2-D, five-point stencil in parallel.
!   - Each processor needs to insert only elements that it owns
!     locally (but any non-local elements will be sent to the
!     appropriate processor during matrix assembly).
!   - Always specify global row and columns of matrix entries.
!   - Note that MatSetValues() uses 0-based row and column numbers
!     in Fortran as well as in C.

!     Note: this uses the less common natural ordering that orders first
!     all the unknowns for x = h then for x = 2h etc; Hence you see JH = II +- n
!     instead of JJ = II +- m as you might expect. The more standard ordering
!     would first do all variables for y = h, then y = 2h etc.

      do 10, II=Istart,Iend-1
        v = -1.0
        i = II/n
        j = II - i*n
        if (i.gt.0) then
          JJ = II - n
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (i.lt.m-1) then
          JJ = II + n
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (j.gt.0) then
          JJ = II - 1
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        if (j.lt.n-1) then
          JJ = II + 1
          call MatSetValues(A,ione,II,ione,JJ,v,INSERT_VALUES,ierr)
        endif
        v = 4.0
        call  MatSetValues(A,ione,II,ione,II,v,INSERT_VALUES,ierr)
 10   continue

!  Assemble matrix, using the 2-step process:
!       MatAssemblyBegin(), MatAssemblyEnd()
!  Computations can be done while messages are in transition,
!  by placing code between these two statements.

      call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
      call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)

!  Create parallel vectors.
!   - Here, the parallel partitioning of the vector is determined by
!     PETSc at runtime.  We could also specify the local dimensions
!     if desired -- or use the more general routine VecCreate().
!   - When solving a linear system, the vectors and matrices MUST
!     be partitioned accordingly.  PETSc automatically generates
!     appropriately partitioned matrices and vectors when MatCreate()
!     and VecCreate() are used with the same communicator.
!   - Note: We form 1 vector from scratch and then duplicate as needed.

      call VecCreateMPI(Petsc_Comm_World,PETSC_DECIDE,m*n,u,ierr)
      call VecSetFromOptions(u,ierr)
      call VecDuplicate(u,b,ierr)
      call VecDuplicate(b,x,ierr)

!  Set exact solution; then compute right-hand-side vector.
!  By default we use an exact solution of a vector with all
!  elements of 1.0;  Alternatively, using the runtime option
!  -random_sol forms a solution vector with random components.

      call PetscOptionsHasName(Petsc_Null_Character,                    &
     &             "-random_exact_sol",flg,ierr)
      if (flg) then
         call PetscRandomCreate(Petsc_Comm_World,rctx,ierr)
         call PetscRandomSetFromOptions(rctx,ierr)
         call VecSetRandom(u,rctx,ierr)
         call PetscRandomDestroy(rctx,ierr)
      else
         call VecSet(u,one,ierr)
      endif
      call MatMult(A,u,b,ierr)

!  View the exact solution vector if desired

      call PetscOptionsHasName(Petsc_Null_Character,                    &
     &             "-view_exact_sol",flg,ierr)
      if (flg) then
         call VecView(u,PETSC_VIEWER_STDOUT_WORLD,ierr)
      endif

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!         Create the linear solver and set various options
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Create linear solver context

      call KSPCreate(Petsc_Comm_World,ksp,ierr)

!  Set operators. Here the matrix that defines the linear system
!  also serves as the preconditioning matrix.

      call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)

!  Set linear solver defaults for this problem (optional).
!   - By extracting the KSP and PC contexts from the KSP context,
!     we can then directly directly call any KSP and PC routines
!     to set various options.
!   - The following four statements are optional; all of these
!     parameters could alternatively be specified at runtime via
!     KSPSetFromOptions(). All of these defaults can be
!     overridden at runtime, as indicated below.

!     We comment out this section of code since the Jacobi
!     preconditioner is not a good general default.

!      call KSPGetPC(ksp,pc,ierr)
!      ptype = PCJACOBI
!      call PCSetType(pc,ptype,ierr)
!      tol = 1.e-7
!      call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,
!     &     PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_INTEGER,ierr)

!  Set user-defined monitoring routine if desired

      call PetscOptionsHasName(Petsc_Null_Character,'-my_ksp_monitor',  &
     &                    flg,ierr)
      if (flg) then
        call KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL_OBJECT,          &
     &                     PETSC_NULL_FUNCTION,ierr)
      endif


!  Set runtime options, e.g.,
!      -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
!  These options will override those specified above as long as
!  KSPSetFromOptions() is called _after_ any other customization
!  routines.

      call KSPSetFromOptions(ksp,ierr)

!  Set convergence test routine if desired

      call PetscOptionsHasName(Petsc_Null_Character,                    &
     &     '-my_ksp_convergence',flg,ierr)
      if (flg) then
        call KSPSetConvergenceTest(ksp,MyKSPConverged,                  &
     &          PETSC_NULL_OBJECT,PETSC_NULL_FUNCTION,ierr)
      endif
!
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                      Solve the linear system
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

      call KSPSolve(ksp,b,x,ierr)

! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
!                     Check solution and clean up
! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

!  Check the error
      call VecAXPY(x,neg_one,u,ierr)
      call VecNorm(x,NORM_2,norm,ierr)
      call KSPGetIterationNumber(ksp,its,ierr)
      if (rank .eq. 0) then
        if (norm .gt. 1.e-12) then
           write(6,100) norm,its
        else
           write(6,110) its
        endif
      endif
  100 format('Norm of error ',e11.4,' iterations ',i5)
  110 format('Norm of error < 1.e-12,iterations ',i5)

!  Free work space.  All PETSc objects should be destroyed when they
!  are no longer needed.

      call KSPDestroy(ksp,ierr)
      call VecDestroy(u,ierr)
      call VecDestroy(x,ierr)
      call VecDestroy(b,ierr)
      call MatDestroy(A,ierr)

!  Always call PetscFinalize() before exiting a program.  This routine
!    - finalizes the PETSc libraries as well as MPI
!    - provides summary and diagnostic information if certain runtime
!      options are chosen (e.g., -log_summary).  See PetscFinalize()
!      manpage for more information.

      call PetscFinalize(ierr)
      end

! --------------------------------------------------------------
!
!  MyKSPMonitor - This is a user-defined routine for monitoring
!  the KSP iterative solvers.
!
!  Input Parameters:
!    ksp   - iterative context
!    n     - iteration number
!    rnorm - 2-norm (preconditioned) residual value (may be estimated)
!    dummy - optional user-defined monitor context (unused here)
!
      subroutine MyKSPMonitor(ksp,n,rnorm,dummy,ierr)

      implicit none

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscksp.h>

      KSP              ksp
      Vec              x
      PetscErrorCode ierr
      PetscInt n,dummy
      PetscMPIInt rank
      double precision rnorm

!  Build the solution vector

      call KSPBuildSolution(ksp,PETSC_NULL_OBJECT,x,ierr)

!  Write the solution vector and residual norm to stdout
!   - Note that the parallel viewer PETSC_VIEWER_STDOUT_WORLD
!     handles data from multiple processors so that the
!     output is not jumbled.

      call MPI_Comm_rank(Petsc_Comm_World,rank,ierr)
      if (rank .eq. 0) write(6,100) n
      call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
      if (rank .eq. 0) write(6,200) n,rnorm

 100  format('iteration ',i5,' solution vector:')
 200  format('iteration ',i5,' residual norm ',e11.4)
      ierr = 0
      end

! --------------------------------------------------------------
!
!  MyKSPConverged - This is a user-defined routine for testing
!  convergence of the KSP iterative solvers.
!
!  Input Parameters:
!    ksp   - iterative context
!    n     - iteration number
!    rnorm - 2-norm (preconditioned) residual value (may be estimated)
!    dummy - optional user-defined monitor context (unused here)
!
      subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)

      implicit none

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscksp.h>

      KSP              ksp
      PetscErrorCode ierr
      PetscInt n,dummy
      KSPConvergedReason flag
      double precision rnorm

      if (rnorm .le. .05) then
        flag = 1
      else
        flag = 0
      endif
      ierr = 0

      end



-------------- next part --------------
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

Petsc-windows-ex2f.exe on a arch-mswin-c-opt named STARGAZER2012 with 1 processor, by danyang Thu Oct 31 16:12:46 2013
With 4 threads per MPI_Comm
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           8.896e-002      1.00000   8.896e-002
Objects:              4.500e+001      1.00000   4.500e+001
Flops:                5.352e+007      1.00000   5.352e+007  5.352e+007
Flops/sec:            6.016e+008      1.00000   6.016e+008  6.016e+008
MPI Messages:         0.000e+000      0.00000   0.000e+000  0.000e+000
MPI Message Lengths:  0.000e+000      0.00000   0.000e+000  0.000e+000
MPI Reductions:       1.410e+002      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 8.8951e-002 100.0%  5.3519e+007 100.0%  0.000e+000   0.0%  0.000e+000        0.0%  1.400e+002  99.3% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult               68 1.0 1.3164e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 15 11  0  0  0  15 11  0  0  0   461
MatSolve              68 1.0 2.1107e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 24 11  0  0  0  24 11  0  0  0   287
MatLUFactorNum         1 1.0 1.5468e-003 1.0 1.09e+005 1.0 0.0e+000 0.0e+000 0.0e+000  2  0  0  0  0   2  0  0  0  0    70
MatILUFactorSym        1 1.0 8.8292e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 1.0e+000  1  0  0  0  1   1  0  0  0  1     0
MatAssemblyBegin       1 1.0 1.1378e-006 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         1 1.0 7.5264e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 1.0 2.8444e-006 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.9911e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
VecMDot               65 1.0 1.1031e-002 1.0 1.89e+007 1.0 0.0e+000 0.0e+000 6.5e+001 12 35  0  0 46  12 35  0  0 46  1713
VecNorm               69 1.0 4.4828e-003 1.0 1.38e+006 1.0 0.0e+000 0.0e+000 6.9e+001  5  3  0  0 49   5  3  0  0 49   308
VecScale              68 1.0 1.3096e-003 1.0 6.80e+005 1.0 0.0e+000 0.0e+000 0.0e+000  1  1  0  0  0   1  1  0  0  0   519
VecCopy                3 1.0 5.1769e-005 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecSet                 5 1.0 6.5991e-005 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                6 1.0 2.9753e-004 1.0 1.20e+005 1.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0   403
VecMAXPY              68 1.0 1.6623e-002 1.0 2.02e+007 1.0 0.0e+000 0.0e+000 0.0e+000 19 38  0  0  0  19 38  0  0  0  1215
VecNormalize          68 1.0 5.8857e-003 1.0 2.04e+006 1.0 0.0e+000 0.0e+000 6.8e+001  7  4  0  0 48   7  4  0  0 49   347
KSPGMRESOrthog        65 1.0 2.6701e-002 1.0 3.78e+007 1.0 0.0e+000 0.0e+000 6.5e+001 30 71  0  0 46  30 71  0  0 46  1416
KSPSetUp               1 1.0 3.7205e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 7.3189e-002 1.0 5.34e+007 1.0 0.0e+000 0.0e+000 1.4e+002 82100  0  0 96  82100  0  0 97   729
PCSetUp                1 1.0 2.6908e-003 1.0 1.09e+005 1.0 0.0e+000 0.0e+000 3.0e+000  3  0  0  0  2   3  0  0  0  2    40
PCApply               68 1.0 2.1146e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 24 11  0  0  0  24 11  0  0  0   287
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix     2              2      1520524     0
              Vector    37             37      3017128     0
       Krylov Solver     1              1        18360     0
      Preconditioner     1              1          976     0
           Index Set     3              3        42280     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 5.68889e-008
#PETSc Option Table entries:
-log_summary log_100x100_openmp_p4.log
-m 100
-n 100
-threadcomm_nthreads 4
-threadcomm_type openmp
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Wed Oct  2 16:35:54 2013
Configure options: --with-cc="win32fe icl" --with-cxx="win32fe icl" --with-fc="win32fe ifort" --with-blas-lapack-dir=/cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64 --with-mpi-include=/cygdrive/c/MSMPI/Inc -with-mpi-lib="[/cygdrive/C/MSMPI/Lib/amd64/msmpi.lib,/cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib]" --with-openmp --with-shared-libraries --with-debugging=no --useThreads=0
-----------------------------------------
Libraries compiled on Wed Oct  2 16:35:54 2013 on NB-TT-113812 
Machine characteristics: CYGWIN_NT-6.1-WOW64-1.7.25-0.270-5-3-i686-32bit
Using PETSc directory: /cygdrive/d/WorkDir/petsc-3.4.2
Using PETSc arch: arch-mswin-c-opt
-----------------------------------------

Using C compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl  -MT -O3 -QxW -Qopenmp  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort  -MT -O3 -QxW -fpp -Qopenmp  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/c/MSMPI/Inc
-----------------------------------------

Using C linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl
Using Fortran linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort
Using libraries: -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -lpetsc /cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64/mkl_intel_lp64_dll.lib mkl_intel_thread_dll.lib mkl_core_dll.lib libiomp5md.lib /cygdrive/C/MSMPI/Lib/amd64/msmpi.lib /cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib 
-----------------------------------------

-------------- next part --------------
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

Petsc-windows-ex2f.exe on a arch-mswin-c-opt named STARGAZER2012 with 1 processor, by danyang Thu Oct 31 16:12:57 2013
With 8 threads per MPI_Comm
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           8.634e-002      1.00000   8.634e-002
Objects:              4.500e+001      1.00000   4.500e+001
Flops:                5.352e+007      1.00000   5.352e+007  5.352e+007
Flops/sec:            6.198e+008      1.00000   6.198e+008  6.198e+008
MPI Messages:         0.000e+000      0.00000   0.000e+000  0.000e+000
MPI Message Lengths:  0.000e+000      0.00000   0.000e+000  0.000e+000
MPI Reductions:       1.410e+002      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 8.6338e-002 100.0%  5.3519e+007 100.0%  0.000e+000   0.0%  0.000e+000        0.0%  1.400e+002  99.3% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult               68 1.0 1.3230e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 15 11  0  0  0  15 11  0  0  0   458
MatSolve              68 1.0 2.0949e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 24 11  0  0  0  24 11  0  0  0   290
MatLUFactorNum         1 1.0 1.5417e-003 1.0 1.09e+005 1.0 0.0e+000 0.0e+000 0.0e+000  2  0  0  0  0   2  0  0  0  0    70
MatILUFactorSym        1 1.0 9.4436e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 1.0e+000  1  0  0  0  1   1  0  0  0  1     0
MatAssemblyBegin       1 1.0 5.6889e-007 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         1 1.0 7.5776e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0     0
MatGetRowIJ            1 1.0 2.8444e-006 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.7579e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
VecMDot               65 1.0 1.0993e-002 1.0 1.89e+007 1.0 0.0e+000 0.0e+000 6.5e+001 13 35  0  0 46  13 35  0  0 46  1719
VecNorm               69 1.0 3.6978e-003 1.0 1.38e+006 1.0 0.0e+000 0.0e+000 6.9e+001  4  3  0  0 49   4  3  0  0 49   373
VecScale              68 1.0 1.0667e-003 1.0 6.80e+005 1.0 0.0e+000 0.0e+000 0.0e+000  1  1  0  0  0   1  1  0  0  0   637
VecCopy                3 1.0 5.0631e-005 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecSet                 5 1.0 6.2009e-005 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                6 1.0 1.4108e-004 1.0 1.20e+005 1.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0   851
VecMAXPY              68 1.0 1.6730e-002 1.0 2.02e+007 1.0 0.0e+000 0.0e+000 0.0e+000 19 38  0  0  0  19 38  0  0  0  1207
VecNormalize          68 1.0 4.8583e-003 1.0 2.04e+006 1.0 0.0e+000 0.0e+000 6.8e+001  6  4  0  0 48   6  4  0  0 49   420
KSPGMRESOrthog        65 1.0 2.6769e-002 1.0 3.78e+007 1.0 0.0e+000 0.0e+000 6.5e+001 31 71  0  0 46  31 71  0  0 46  1412
KSPSetUp               1 1.0 3.2484e-004 1.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 7.1967e-002 1.0 5.34e+007 1.0 0.0e+000 0.0e+000 1.4e+002 83100  0  0 96  83100  0  0 97   742
PCSetUp                1 1.0 2.7182e-003 1.0 1.09e+005 1.0 0.0e+000 0.0e+000 3.0e+000  3  0  0  0  2   3  0  0  0  2    40
PCApply               68 1.0 2.0985e-002 1.0 6.07e+006 1.0 0.0e+000 0.0e+000 0.0e+000 24 11  0  0  0  24 11  0  0  0   289
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix     2              2      1520524     0
              Vector    37             37      3017128     0
       Krylov Solver     1              1        18360     0
      Preconditioner     1              1          976     0
           Index Set     3              3        42280     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 1.13778e-007
#PETSc Option Table entries:
-log_summary log_100x100_openmp_p8.log
-m 100
-n 100
-threadcomm_nthreads 8
-threadcomm_type openmp
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Wed Oct  2 16:35:54 2013
Configure options: --with-cc="win32fe icl" --with-cxx="win32fe icl" --with-fc="win32fe ifort" --with-blas-lapack-dir=/cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64 --with-mpi-include=/cygdrive/c/MSMPI/Inc -with-mpi-lib="[/cygdrive/C/MSMPI/Lib/amd64/msmpi.lib,/cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib]" --with-openmp --with-shared-libraries --with-debugging=no --useThreads=0
-----------------------------------------
Libraries compiled on Wed Oct  2 16:35:54 2013 on NB-TT-113812 
Machine characteristics: CYGWIN_NT-6.1-WOW64-1.7.25-0.270-5-3-i686-32bit
Using PETSc directory: /cygdrive/d/WorkDir/petsc-3.4.2
Using PETSc arch: arch-mswin-c-opt
-----------------------------------------

Using C compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl  -MT -O3 -QxW -Qopenmp  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort  -MT -O3 -QxW -fpp -Qopenmp  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/c/MSMPI/Inc
-----------------------------------------

Using C linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl
Using Fortran linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort
Using libraries: -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -lpetsc /cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64/mkl_intel_lp64_dll.lib mkl_intel_thread_dll.lib mkl_core_dll.lib libiomp5md.lib /cygdrive/C/MSMPI/Lib/amd64/msmpi.lib /cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib 
-----------------------------------------

-------------- next part --------------
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

Petsc-windows-ex2f.exe on a arch-mswin-c-opt named STARGAZER2012 with 4 processors, by danyang Thu Oct 31 16:08:41 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           6.401e-002      1.01274   6.350e-002
Objects:              5.600e+001      1.00000   5.600e+001
Flops:                2.031e+007      1.00102   2.030e+007  8.120e+007
Flops/sec:            3.213e+008      1.01377   3.197e+008  1.279e+009
MPI Messages:         2.100e+002      2.00000   1.575e+002  6.300e+002
MPI Message Lengths:  1.656e+005      2.00000   7.886e+002  4.968e+005
MPI Reductions:       2.270e+002      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.3490e-002 100.0%  8.1202e+007 100.0%  6.300e+002 100.0%  7.886e+002      100.0%  2.260e+002  99.6% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult              103 1.0 1.3790e-002 1.9 2.31e+006 1.0 6.2e+002 8.0e+002 0.0e+000 17 11 98100  0  17 11 98100  0   666
MatSolve             103 1.0 1.1147e-002 1.4 2.27e+006 1.0 0.0e+000 0.0e+000 0.0e+000 16 11  0  0  0  16 11  0  0  0   813
MatLUFactorNum         1 1.0 3.9652e-004 1.0 2.66e+004 1.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0   269
MatILUFactorSym        1 1.0 2.7420e-004 1.1 0.00e+000 0.0 0.0e+000 0.0e+000 1.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin       1 1.0 1.1662e-004 1.3 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
MatAssemblyEnd         1 1.0 1.2538e-003 1.0 0.00e+000 0.0 1.2e+001 2.0e+002 9.0e+000  2  0  2  0  4   2  0  2  0  4     0
MatGetRowIJ            1 1.0 7.3956e-006 2.6 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 7.1680e-005 1.2 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
VecMDot               99 1.0 1.5917e-002 2.0 7.20e+006 1.0 0.0e+000 0.0e+000 9.9e+001 17 35  0  0 44  17 35  0  0 44  1809
VecNorm              104 1.0 9.6899e-003 4.3 5.20e+005 1.0 0.0e+000 0.0e+000 1.0e+002  8  3  0  0 46   8  3  0  0 46   215
VecScale             103 1.0 4.1813e-004 1.6 2.58e+005 1.0 0.0e+000 0.0e+000 0.0e+000  1  1  0  0  0   1  1  0  0  0  2463
VecCopy                4 1.0 4.2667e-005 1.5 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecSet               110 1.0 4.7957e-004 1.5 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0     0
VecAXPY                8 1.0 9.3008e-003 1.5 4.00e+004 1.0 0.0e+000 0.0e+000 0.0e+000 11  0  0  0  0  11  0  0  0  0    17
VecMAXPY             103 1.0 1.0259e-002 1.6 7.70e+006 1.0 0.0e+000 0.0e+000 0.0e+000 14 38  0  0  0  14 38  0  0  0  3000
VecScatterBegin      103 1.0 6.1099e-004 1.7 0.00e+000 0.0 6.2e+002 8.0e+002 0.0e+000  1  0 98100  0   1  0 98100  0     0
VecScatterEnd        103 1.0 3.1807e-003 7.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  2  0  0  0  0   2  0  0  0  0     0
VecNormalize         103 1.0 9.9965e-003 3.6 7.73e+005 1.0 0.0e+000 0.0e+000 1.0e+002  9  4  0  0 45   9  4  0  0 46   309
KSPGMRESOrthog        99 1.0 2.1869e-002 1.2 1.44e+007 1.0 0.0e+000 0.0e+000 9.9e+001 30 71  0  0 44  30 71  0  0 44  2634
KSPSetUp               2 1.0 1.4677e-004 1.1 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 5.7238e-002 1.0 2.03e+007 1.0 6.1e+002 8.0e+002 2.1e+002 90100 97 99 91  90100 97 99 92  1416
PCSetUp                2 1.0 9.2729e-004 1.0 2.66e+004 1.0 0.0e+000 0.0e+000 5.0e+000  1  0  0  0  2   1  0  0  0  2   115
PCSetUpOnBlocks        1 1.0 7.8507e-004 1.0 2.66e+004 1.0 0.0e+000 0.0e+000 3.0e+000  1  0  0  0  1   1  0  0  0  1   136
PCApply              103 1.0 1.3624e-002 1.4 2.27e+006 1.0 0.0e+000 0.0e+000 0.0e+000 19 11  0  0  0  19 11  0  0  0   665
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix     4              4       577836     0
              Vector    41             41       803912     0
      Vector Scatter     1              1         1052     0
           Index Set     5              5        14192     0
       Krylov Solver     2              2        19504     0
      Preconditioner     2              2         1848     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 5.68889e-008
Average time for MPI_Barrier(): 2.38933e-006
Average time for zero size MPI_Send(): 2.13333e-006
#PETSc Option Table entries:
-log_summary log_100x100_mpi_p4.log
-m 100
-n 100
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Wed Oct  2 16:35:54 2013
Configure options: --with-cc="win32fe icl" --with-cxx="win32fe icl" --with-fc="win32fe ifort" --with-blas-lapack-dir=/cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64 --with-mpi-include=/cygdrive/c/MSMPI/Inc -with-mpi-lib="[/cygdrive/C/MSMPI/Lib/amd64/msmpi.lib,/cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib]" --with-openmp --with-shared-libraries --with-debugging=no --useThreads=0
-----------------------------------------
Libraries compiled on Wed Oct  2 16:35:54 2013 on NB-TT-113812 
Machine characteristics: CYGWIN_NT-6.1-WOW64-1.7.25-0.270-5-3-i686-32bit
Using PETSc directory: /cygdrive/d/WorkDir/petsc-3.4.2
Using PETSc arch: arch-mswin-c-opt
-----------------------------------------

Using C compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl  -MT -O3 -QxW -Qopenmp  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort  -MT -O3 -QxW -fpp -Qopenmp  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/c/MSMPI/Inc
-----------------------------------------

Using C linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl
Using Fortran linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort
Using libraries: -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -lpetsc /cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64/mkl_intel_lp64_dll.lib mkl_intel_thread_dll.lib mkl_core_dll.lib libiomp5md.lib /cygdrive/C/MSMPI/Lib/amd64/msmpi.lib /cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib 
-----------------------------------------

-------------- next part --------------
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

Petsc-windows-ex2f.exe on a arch-mswin-c-opt named STARGAZER2012 with 8 processors, by danyang Thu Oct 31 16:08:44 2013
Using Petsc Release Version 3.4.2, Jul, 02, 2013 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.930e-002      1.03041   2.877e-002
Objects:              5.600e+001      1.00000   5.600e+001
Flops:                1.090e+007      1.00204   1.089e+007  8.715e+007
Flops/sec:            3.833e+008      1.03251   3.787e+008  3.030e+009
MPI Messages:         2.260e+002      2.00000   1.978e+002  1.582e+003
MPI Message Lengths:  1.784e+005      2.00000   7.894e+002  1.249e+006
MPI Reductions:       2.430e+002      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.8748e-002  99.9%  8.7151e+007 100.0%  1.582e+003 100.0%  7.894e+002      100.0%  2.420e+002  99.6% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %f - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult              111 1.0 4.7565e-003 1.1 1.24e+006 1.0 1.6e+003 8.0e+002 0.0e+000 16 11 98100  0  16 11 98100  0  2082
MatSolve             111 1.0 4.3031e-003 1.0 1.20e+006 1.0 0.0e+000 0.0e+000 0.0e+000 15 11  0  0  0  15 11  0  0  0  2228
MatLUFactorNum         1 1.0 2.0708e-004 1.1 1.30e+004 1.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0   501
MatILUFactorSym        1 1.0 1.5815e-004 1.1 0.00e+000 0.0 0.0e+000 0.0e+000 1.0e+000  1  0  0  0  0   1  0  0  0  0     0
MatAssemblyBegin       1 1.0 1.4336e-004 1.3 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
MatAssemblyEnd         1 1.0 1.7192e-003 1.0 0.00e+000 0.0 2.8e+001 2.0e+002 9.0e+000  6  0  2  0  4   6  0  2  0  4     0
MatGetRowIJ            1 1.0 4.5511e-006 2.0 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 4.7787e-005 1.1 0.00e+000 0.0 0.0e+000 0.0e+000 2.0e+000  0  0  0  0  1   0  0  0  0  1     0
VecMDot              107 1.0 4.4311e-003 1.2 3.87e+006 1.0 0.0e+000 0.0e+000 1.1e+002 14 36  0  0 44  14 36  0  0 44  6984
VecNorm              112 1.0 3.2262e-003 1.0 2.80e+005 1.0 0.0e+000 0.0e+000 1.1e+002 11  3  0  0 46  11  3  0  0 46   694
VecScale             111 1.0 2.1390e-004 1.2 1.39e+005 1.0 0.0e+000 0.0e+000 0.0e+000  1  1  0  0  0   1  1  0  0  0  5189
VecCopy                4 1.0 4.4942e-005 3.2 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
VecSet               118 1.0 2.0594e-004 1.2 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  1  0  0  0  0   1  0  0  0  0     0
VecAXPY                8 1.0 6.4853e-005 1.8 2.00e+004 1.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0  2467
VecMAXPY             111 1.0 3.4446e-003 1.0 4.14e+006 1.0 0.0e+000 0.0e+000 0.0e+000 12 38  0  0  0  12 38  0  0  0  9609
VecScatterBegin      111 1.0 5.5694e-004 1.7 0.00e+000 0.0 1.6e+003 8.0e+002 0.0e+000  2  0 98100  0   2  0 98100  0     0
VecScatterEnd        111 1.0 5.4556e-004 1.7 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  2  0  0  0  0   2  0  0  0  0     0
VecNormalize         111 1.0 3.5487e-003 1.0 4.16e+005 1.0 0.0e+000 0.0e+000 1.1e+002 12  4  0  0 46  12  4  0  0 46   938
KSPGMRESOrthog       107 1.0 7.7466e-003 1.1 7.74e+006 1.0 0.0e+000 0.0e+000 1.1e+002 26 71  0  0 44  26 71  0  0 44  7992
KSPSetUp               2 1.0 1.7180e-004 1.6 0.00e+000 0.0 0.0e+000 0.0e+000 0.0e+000  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 2.3104e-002 1.0 1.09e+007 1.0 1.5e+003 8.0e+002 2.2e+002 80100 97 99 92  80100 97 99 92  3766
PCSetUp                2 1.0 6.2976e-004 1.1 1.30e+004 1.0 0.0e+000 0.0e+000 5.0e+000  2  0  0  0  2   2  0  0  0  2   165
PCSetUpOnBlocks        1 1.0 4.5852e-004 1.1 1.30e+004 1.0 0.0e+000 0.0e+000 3.0e+000  2  0  0  0  1   2  0  0  0  1   226
PCApply              111 1.0 5.9147e-003 1.0 1.20e+006 1.0 0.0e+000 0.0e+000 0.0e+000 21 11  0  0  0  21 11  0  0  0  1621
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Matrix     4              4       293124     0
              Vector    41             41       433912     0
      Vector Scatter     1              1         1052     0
           Index Set     5              5         9192     0
       Krylov Solver     2              2        19504     0
      Preconditioner     2              2         1848     0
              Viewer     1              0            0     0
========================================================================================================================
Average time to get PetscTime(): 5.68889e-008
Average time for MPI_Barrier(): 5.00622e-006
Average time for zero size MPI_Send(): 2.27556e-006
#PETSc Option Table entries:
-log_summary log_100x100_mpi_p8.log
-m 100
-n 100
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 4 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure run at: Wed Oct  2 16:35:54 2013
Configure options: --with-cc="win32fe icl" --with-cxx="win32fe icl" --with-fc="win32fe ifort" --with-blas-lapack-dir=/cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64 --with-mpi-include=/cygdrive/c/MSMPI/Inc -with-mpi-lib="[/cygdrive/C/MSMPI/Lib/amd64/msmpi.lib,/cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib]" --with-openmp --with-shared-libraries --with-debugging=no --useThreads=0
-----------------------------------------
Libraries compiled on Wed Oct  2 16:35:54 2013 on NB-TT-113812 
Machine characteristics: CYGWIN_NT-6.1-WOW64-1.7.25-0.270-5-3-i686-32bit
Using PETSc directory: /cygdrive/d/WorkDir/petsc-3.4.2
Using PETSc arch: arch-mswin-c-opt
-----------------------------------------

Using C compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl  -MT -O3 -QxW -Qopenmp  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort  -MT -O3 -QxW -fpp -Qopenmp  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/include -I/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/include -I/cygdrive/c/MSMPI/Inc
-----------------------------------------

Using C linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe icl
Using Fortran linker: /cygdrive/d/WorkDir/petsc-3.4.2/bin/win32fe/win32fe ifort
Using libraries: -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -L/cygdrive/d/WorkDir/petsc-3.4.2/arch-mswin-c-opt/lib -lpetsc /cygdrive/d/HardLinks/PETSc/Intel2013/mkl/lib/intel64/mkl_intel_lp64_dll.lib mkl_intel_thread_dll.lib mkl_core_dll.lib libiomp5md.lib /cygdrive/C/MSMPI/Lib/amd64/msmpi.lib /cygdrive/C/MSMPI/Lib/amd64/msmpifec.lib Gdi32.lib User32.lib Advapi32.lib Kernel32.lib Ws2_32.lib 
-----------------------------------------


From torquil at gmail.com  Thu Oct 31 19:45:59 2013
From: torquil at gmail.com (=?ISO-8859-1?Q?Torquil_Macdonald_S=F8rensen?=)
Date: Fri, 01 Nov 2013 01:45:59 +0100
Subject: [petsc-users] ILUTP
In-Reply-To: <CAFvbobWEvYfB+X65UeE1vMN=ocCpvd0iV++y-tc_+pUVrV55pw@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
	<CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
	<CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>
	<CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>
	<CAFvbobWEvYfB+X65UeE1vMN=ocCpvd0iV++y-tc_+pUVrV55pw@mail.gmail.com>
Message-ID: <5272F9C7.7030000@gmail.com>

On 30/10/13 19:01, Xiaoye S. Li wrote:
> The sequential SuperLU has ILUTP implementation, not in parallel
> versions. PETSc already supports the option of using SuperLU. 
>
> In SuperLU distribution:
>   EXAMPLE/zitersol.c : an example to use GMRES with ILUTP
> preconditioner (returned from driver SRC/zgsisx.c)
>   SRC/zgsitrf.c : the actual ILUTP factorization routine
>
> Sherry Li
>

Thanks! I installed the sequential SuperLU and managed to get it working
with PETSc. I also managed to get convergence with the ASM
preconditioner, but ILUTP is much faster, and I can use it when I'm
doing uniprocessor runs. It is convenient that I don't have to keep a
separate program version for those cases. Now I can do PETSc-based
solving in all cases, and relegate GetFEM/GMM to only finite element
matrix assembly.

It was a bit of a hassle to get a working install of both PETSc and
GetFEM due to SuperLU-related linking problems. The built-in SuperLU in
GetFEM caused problems when linking a program to both GetFEM and PETSc.
But in the end I managed to conjure up a SuperLU shared library, and get
both GetFEM and PETSc to use that.

Best regards
Torquil S?rensen


From torquil at gmail.com  Thu Oct 31 19:53:52 2013
From: torquil at gmail.com (=?ISO-8859-1?Q?Torquil_Macdonald_S=F8rensen?=)
Date: Fri, 01 Nov 2013 01:53:52 +0100
Subject: [petsc-users] ILUTP
In-Reply-To: <CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>
References: <CAD6PAKPmUpjSUd7i+QjRZjKFB3oVpE_1gL9Wmbs2jugEf3YXcA@mail.gmail.com>
	<CAMYG4Gn+B_XmOoGHoeShSr7+H+8vwmwrmts4ESFUU6hOPR8k9g@mail.gmail.com>
	<CAD6PAKMRSOjYQChUis+fMY_gyX0moS_wKa99Wi-52hk2LxaPSA@mail.gmail.com>
	<CAMYG4Gm507y-6XaENFzeAwnSki-Vp7zhaNCxcu=7dMSnODJMeA@mail.gmail.com>
Message-ID: <5272FBA0.3080609@gmail.com>

On 30/10/13 16:48, Matthew Knepley wrote:
> On Wed, Oct 30, 2013 at 8:58 AM, Torquil Macdonald S?rensen
> <torquil at gmail.com <mailto:torquil at gmail.com>> wrote:
>
>     Thanks Matthew!
>
>     The problem I'm working on is the Dirac equation, in various
>     number of dimensions. I'm going to take a stab at multigrid
>     preconditioning.
>
>
> My limited knowledge of the Dirac Equation tells me that it is a
> relativistic wave equation, and thus hyperbolic, which is quite
> difficult for
> MG (but doable with a bunch of work). How do you have this formulated?
>

I'm doing time-stepping at the moment, so I'm solving an elliptic
equation at each time step. No fancy hyperbolic multigrid work :-)

Best regards
Torquil S?rensen

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131101/9dcf6313/attachment.html>

From jbermeol at purdue.edu  Thu Oct 31 20:38:43 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Thu, 31 Oct 2013 21:38:43 -0400 (EDT)
Subject: [petsc-users] [petsc-dev] MatCreateDense
In-Reply-To: <CAMYG4GmwCv4Rv11eJ-dEF0NmOGXO_zKn8=b1yv45UTBqkLMWdg@mail.gmail.com>
Message-ID: <629141832.347605.1383269923105.JavaMail.root@mailhub059.itcs.purdue.edu>

What do you mean with number of local columns??

Thanks

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "petsc-dev" <petsc-dev at mcs.anl.gov>, "petsc-users" <petsc-users at mcs.anl.gov>
Sent: Thursday, October 31, 2013 2:37:43 PM
Subject: Re: [petsc-dev] MatCreateDense


On Thu, Oct 31, 2013 at 1:21 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 




Hi small question. In the method MatCreateSeqDense(MPI_Comm comm,PetscInt m,PetscInt n,PetscScalar *data,Mat *A) I'm giving the local number of rows and columns, so I should pass the total number of local columns or I should pass columns/number_of_MPI process. 



If its sequential, you pass the total number of columns. If its MatCreateMPIDense(), pass the number of local columns. This 
is used to create a vector with the same layout as y = A^T x. 


Matt 


Thanks 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From jbermeol at purdue.edu  Thu Oct 31 20:41:37 2013
From: jbermeol at purdue.edu (Jose David Bermeol)
Date: Thu, 31 Oct 2013 21:41:37 -0400 (EDT)
Subject: [petsc-users] [petsc-dev] MatCreateDense
In-Reply-To: <629141832.347605.1383269923105.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <1844528311.347613.1383270097148.JavaMail.root@mailhub059.itcs.purdue.edu>

For instance I have a matrix of 400x8 and I want to split it in two mpi process, one with the first 100 rows and the second one with the next 300 rows.

----- Original Message -----
From: "Jose David Bermeol" <jbermeol at purdue.edu>
To: "Matthew Knepley" <knepley at gmail.com>
Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "petsc-dev" <petsc-dev at mcs.anl.gov>
Sent: Thursday, October 31, 2013 9:38:43 PM
Subject: Re: [petsc-dev] MatCreateDense

What do you mean with number of local columns??

Thanks

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "Jose David Bermeol" <jbermeol at purdue.edu>
Cc: "petsc-dev" <petsc-dev at mcs.anl.gov>, "petsc-users" <petsc-users at mcs.anl.gov>
Sent: Thursday, October 31, 2013 2:37:43 PM
Subject: Re: [petsc-dev] MatCreateDense


On Thu, Oct 31, 2013 at 1:21 PM, Jose David Bermeol < jbermeol at purdue.edu > wrote: 




Hi small question. In the method MatCreateSeqDense(MPI_Comm comm,PetscInt m,PetscInt n,PetscScalar *data,Mat *A) I'm giving the local number of rows and columns, so I should pass the total number of local columns or I should pass columns/number_of_MPI process. 



If its sequential, you pass the total number of columns. If its MatCreateMPIDense(), pass the number of local columns. This 
is used to create a vector with the same layout as y = A^T x. 


Matt 


Thanks 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From knepley at gmail.com  Thu Oct 31 21:48:59 2013
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 31 Oct 2013 21:48:59 -0500
Subject: [petsc-users] [petsc-dev] MatCreateDense
In-Reply-To: <1844528311.347613.1383270097148.JavaMail.root@mailhub059.itcs.purdue.edu>
References: <629141832.347605.1383269923105.JavaMail.root@mailhub059.itcs.purdue.edu>
	<1844528311.347613.1383270097148.JavaMail.root@mailhub059.itcs.purdue.edu>
Message-ID: <CAMYG4Gk6U-W6We899CO6-unZut_6whGbmrS8djp4+x5e4x02uA@mail.gmail.com>

On Thu, Oct 31, 2013 at 8:41 PM, Jose David Bermeol <jbermeol at purdue.edu>wrote:

> For instance I have a matrix of 400x8 and I want to split it in two mpi
> process, one with the first 100 rows and the second one with the next 300
> rows.


1) Its not necessary to mail users and dev

2) You prescribed the range space, (100, 300), now you can prescribe the
domain by dividing the columns. It not usually important

   Matt


> ----- Original Message -----
> From: "Jose David Bermeol" <jbermeol at purdue.edu>
> To: "Matthew Knepley" <knepley at gmail.com>
> Cc: "petsc-users" <petsc-users at mcs.anl.gov>, "petsc-dev" <
> petsc-dev at mcs.anl.gov>
> Sent: Thursday, October 31, 2013 9:38:43 PM
> Subject: Re: [petsc-dev] MatCreateDense
>
> What do you mean with number of local columns??
>
> Thanks
>
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "Jose David Bermeol" <jbermeol at purdue.edu>
> Cc: "petsc-dev" <petsc-dev at mcs.anl.gov>, "petsc-users" <
> petsc-users at mcs.anl.gov>
> Sent: Thursday, October 31, 2013 2:37:43 PM
> Subject: Re: [petsc-dev] MatCreateDense
>
>
> On Thu, Oct 31, 2013 at 1:21 PM, Jose David Bermeol < jbermeol at purdue.edu> wrote:
>
>
>
>
> Hi small question. In the method MatCreateSeqDense(MPI_Comm comm,PetscInt
> m,PetscInt n,PetscScalar *data,Mat *A) I'm giving the local number of rows
> and columns, so I should pass the total number of local columns or I should
> pass columns/number_of_MPI process.
>
>
>
> If its sequential, you pass the total number of columns. If its
> MatCreateMPIDense(), pass the number of local columns. This
> is used to create a vector with the same layout as y = A^T x.
>
>
> Matt
>
>
> Thanks
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131031/20bd99ed/attachment.html>