From C.Klaij at marin.nl  Fri May  1 03:06:19 2015
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Fri, 1 May 2015 08:06:19 +0000
Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to
 MatNestGetISs in fortran
In-Reply-To: <alpine.LFD.2.11.1504301452460.2844@asterix>
References: <1429875994856.10134@marin.nl>,
	<alpine.LFD.2.11.1504241543020.4122@asterix>
	<1430388147149.79103@marin.nl>,
	<alpine.LFD.2.11.1504301452460.2844@asterix>
Message-ID: <1430467579805.85425@marin.nl>

Satish,

Today, I tried again and now I also get:

                     0
                     0

Sorry, I must have done something wrong earlier... Thanks again,

Chris

________________________________________
From: Satish Balay <balay at mcs.anl.gov>
Sent: Thursday, April 30, 2015 9:54 PM
To: Klaij, Christiaan
Cc: petsc-users
Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran

Hm - it works for me with the test code you sent..

<petsc-3.5.3 before patch>

balay at asterix /home/balay/download-pine/x
$ ./ex1f
                    0
             40357424

<petsc-3.5.3 after patch>

balay at asterix /home/balay/download-pine/x
$ ./ex1f
                    0
                    0
balay at asterix /home/balay/download-pine/x

Satish


$ On Thu, 30 Apr 2015, Klaij, Christiaan wrote:

> Satish,
>
> Thanks for the files! I've copied them to the correct location in my petsc-3.5.3 dir and rebuild the whole thing from scratch but I still get the same memory corruption for the example below...
>
> Chris
> ________________________________________
> From: Satish Balay <balay at mcs.anl.gov>
> Sent: Friday, April 24, 2015 10:54 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran
>
> Sorry for dropping the ball on this issue. I pushed the following fix to maint branch.
>
> https://bitbucket.org/petsc/petsc/commits/3a4d7b9a6c83003720b45dc0635fc32ea52a4309
>
> To use this change with petsc-3.5.3 - you can drop in the attached replacement files at:
>
> src/mat/impls/nest/ftn-custom/zmatnestf.c
> src/mat/impls/nest/ftn-auto/matnestf.c
>
> Satish
>
> On Fri, 24 Apr 2015, Klaij, Christiaan wrote:
>
> > Barry, Satish
> >
> > Any news on this issue?
> >
> > Chris
> >
> > > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry <bsmith at mcs.anl.gov> wrote:
> > >
> > >    Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument.
> > >
> > >     Satish, could you please see if you can add a custom fortran stub for this function in maint?
> > >
> > >   Thanks
> > >
> > >    Barry
> > >
> > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> > > >
> > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example:
> > > >
> > > > $ cat fieldsplittry2.F90
> > > > program fieldsplittry2
> > > >
> > > >  use petscksp
> > > >  implicit none
> > > > #include <finclude/petsckspdef.h>
> > > >
> > > >  PetscErrorCode :: ierr
> > > >  PetscInt       :: size,i,j,start,end,n=4,numsplit=1
> > > >  PetscScalar    :: zero=0.0,one=1.0
> > > >  Vec            :: diag3,x,b
> > > >  Mat            :: A,subA(4),myS
> > > >  PC             :: pc,subpc(2)
> > > >  KSP            :: ksp,subksp(2)
> > > >  IS             :: isg(2)
> > > >
> > > >  call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr)
> > > >  call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr);
> > > >
> > > >  ! vectors
> > > >  call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr)
> > > >  call VecSet(diag3,one,ierr); CHKERRQ(ierr)
> > > >
> > > >  call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr)
> > > >  call VecSet(x,zero,ierr); CHKERRQ(ierr)
> > > >
> > > >  call VecDuplicate(x,b,ierr); CHKERRQ(ierr)
> > > >  call VecSet(b,one,ierr); CHKERRQ(ierr)
> > > >
> > > >  ! matrix a00
> > > >  call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr)
> > > >  call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >
> > > >  ! matrix a01
> > > >  call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr)
> > > >  call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr);
> > > >  do i=start,end-1
> > > >     j=mod(i,size*n)
> > > >     call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr)
> > > >  end do
> > > >  call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >
> > > >  ! matrix a10
> > > >  call  MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr)
> > > >
> > > >  ! matrix a11 (empty)
> > > >  call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >
> > > >  ! nested mat [a00,a01;a10,a11]
> > > >  call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > >  call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > print *, PETSC_NULL_OBJECT
> > > >  call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr);
> > > > print *, PETSC_NULL_OBJECT
> > > >
> > > >  call PetscFinalize(ierr)
> > > >
> > > > end program fieldsplittry2
> > > > $ ./fieldsplittry2
> > > >                     0
> > > >              39367824
> > > > $
> > > >
> > > >
> > > > dr. ir. Christiaan Klaij
> > > > CFD Researcher
> > > > Research & Development
> > > > E mailto:C.Klaij at marin.nl
> > > > T +31 317 49 33 44
> > > >
> > > >
> > > > MARIN
> > > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
> > > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl
> > > >
> >
> >
> > dr. ir. Christiaan Klaij
> > CFD Researcher
> > Research & Development
> > E mailto:C.Klaij at marin.nl
> > T +31 317 49 33 44
> >
> >
> > MARIN
> > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
> > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl
> >
> >
>


From balay at mcs.anl.gov  Fri May  1 09:15:43 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 1 May 2015 09:15:43 -0500
Subject: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to
 MatNestGetISs in fortran
In-Reply-To: <1430467579805.85425@marin.nl>
References: <1429875994856.10134@marin.nl>,
	<alpine.LFD.2.11.1504241543020.4122@asterix>
	<1430388147149.79103@marin.nl>,
	<alpine.LFD.2.11.1504301452460.2844@asterix>
	<1430467579805.85425@marin.nl>
Message-ID: <alpine.LFD.2.11.1505010915340.22774@asterix>

Great! Glad it works!

Satish

On Fri, 1 May 2015, Klaij, Christiaan wrote:

> Satish,
> 
> Today, I tried again and now I also get:
> 
>                      0
>                      0
> 
> Sorry, I must have done something wrong earlier... Thanks again,
> 
> Chris
> 
> ________________________________________
> From: Satish Balay <balay at mcs.anl.gov>
> Sent: Thursday, April 30, 2015 9:54 PM
> To: Klaij, Christiaan
> Cc: petsc-users
> Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran
> 
> Hm - it works for me with the test code you sent..
> 
> <petsc-3.5.3 before patch>
> 
> balay at asterix /home/balay/download-pine/x
> $ ./ex1f
>                     0
>              40357424
> 
> <petsc-3.5.3 after patch>
> 
> balay at asterix /home/balay/download-pine/x
> $ ./ex1f
>                     0
>                     0
> balay at asterix /home/balay/download-pine/x
> 
> Satish
> 
> 
> $ On Thu, 30 Apr 2015, Klaij, Christiaan wrote:
> 
> > Satish,
> >
> > Thanks for the files! I've copied them to the correct location in my petsc-3.5.3 dir and rebuild the whole thing from scratch but I still get the same memory corruption for the example below...
> >
> > Chris
> > ________________________________________
> > From: Satish Balay <balay at mcs.anl.gov>
> > Sent: Friday, April 24, 2015 10:54 PM
> > To: Klaij, Christiaan
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] PETSC_NULL_OBJECT gets corrupt after call to MatNestGetISs in fortran
> >
> > Sorry for dropping the ball on this issue. I pushed the following fix to maint branch.
> >
> > https://bitbucket.org/petsc/petsc/commits/3a4d7b9a6c83003720b45dc0635fc32ea52a4309
> >
> > To use this change with petsc-3.5.3 - you can drop in the attached replacement files at:
> >
> > src/mat/impls/nest/ftn-custom/zmatnestf.c
> > src/mat/impls/nest/ftn-auto/matnestf.c
> >
> > Satish
> >
> > On Fri, 24 Apr 2015, Klaij, Christiaan wrote:
> >
> > > Barry, Satish
> > >
> > > Any news on this issue?
> > >
> > > Chris
> > >
> > > > On Feb 12, 2015, at 07:13:08 CST, Smith, Barry <bsmith at mcs.anl.gov> wrote:
> > > >
> > > >    Thanks for reporting this. Currently the Fortran stub for this function is generated automatically which means it does not have the logic for handling a PETSC_NULL_OBJECT argument.
> > > >
> > > >     Satish, could you please see if you can add a custom fortran stub for this function in maint?
> > > >
> > > >   Thanks
> > > >
> > > >    Barry
> > > >
> > > > > On Feb 12, 2015, at 3:02 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> > > > >
> > > > > Using petsc-3.5.3, I noticed that PETSC_NULL_OBJECT gets corrupt after calling MatNestGetISs in fortran. Here's a small example:
> > > > >
> > > > > $ cat fieldsplittry2.F90
> > > > > program fieldsplittry2
> > > > >
> > > > >  use petscksp
> > > > >  implicit none
> > > > > #include <finclude/petsckspdef.h>
> > > > >
> > > > >  PetscErrorCode :: ierr
> > > > >  PetscInt       :: size,i,j,start,end,n=4,numsplit=1
> > > > >  PetscScalar    :: zero=0.0,one=1.0
> > > > >  Vec            :: diag3,x,b
> > > > >  Mat            :: A,subA(4),myS
> > > > >  PC             :: pc,subpc(2)
> > > > >  KSP            :: ksp,subksp(2)
> > > > >  IS             :: isg(2)
> > > > >
> > > > >  call PetscInitialize(PETSC_NULL_CHARACTER,ierr); CHKERRQ(ierr)
> > > > >  call MPI_Comm_size(PETSC_COMM_WORLD,size,ierr); CHKERRQ(ierr);
> > > > >
> > > > >  ! vectors
> > > > >  call VecCreateMPI(MPI_COMM_WORLD,3*n,PETSC_DECIDE,diag3,ierr); CHKERRQ(ierr)
> > > > >  call VecSet(diag3,one,ierr); CHKERRQ(ierr)
> > > > >
> > > > >  call VecCreateMPI(MPI_COMM_WORLD,4*n,PETSC_DECIDE,x,ierr); CHKERRQ(ierr)
> > > > >  call VecSet(x,zero,ierr); CHKERRQ(ierr)
> > > > >
> > > > >  call VecDuplicate(x,b,ierr); CHKERRQ(ierr)
> > > > >  call VecSet(b,one,ierr); CHKERRQ(ierr)
> > > > >
> > > > >  ! matrix a00
> > > > >  call MatCreateAIJ(MPI_COMM_WORLD,3*n,3*n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(1),ierr);CHKERRQ(ierr)
> > > > >  call MatDiagonalSet(subA(1),diag3,INSERT_VALUES,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyBegin(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyEnd(subA(1),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >
> > > > >  ! matrix a01
> > > > >  call MatCreateAIJ(MPI_COMM_WORLD,3*n,n,PETSC_DECIDE,PETSC_DECIDE,1,PETSC_NULL_INTEGER,1,PETSC_NULL_INTEGER,subA(2),ierr);CHKERRQ(ierr)
> > > > >  call MatGetOwnershipRange(subA(2),start,end,ierr);CHKERRQ(ierr);
> > > > >  do i=start,end-1
> > > > >     j=mod(i,size*n)
> > > > >     call MatSetValue(subA(2),i,j,one,INSERT_VALUES,ierr);CHKERRQ(ierr)
> > > > >  end do
> > > > >  call MatAssemblyBegin(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyEnd(subA(2),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >
> > > > >  ! matrix a10
> > > > >  call  MatTranspose(subA(2),MAT_INITIAL_MATRIX,subA(3),ierr);CHKERRQ(ierr)
> > > > >
> > > > >  ! matrix a11 (empty)
> > > > >  call MatCreateAIJ(MPI_COMM_WORLD,n,n,PETSC_DECIDE,PETSC_DECIDE,0,PETSC_NULL_INTEGER,0,PETSC_NULL_INTEGER,subA(4),ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyBegin(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyEnd(subA(4),MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >
> > > > >  ! nested mat [a00,a01;a10,a11]
> > > > >  call MatCreateNest(MPI_COMM_WORLD,2,PETSC_NULL_OBJECT,2,PETSC_NULL_OBJECT,subA,A,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > >  call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr);CHKERRQ(ierr)
> > > > > print *, PETSC_NULL_OBJECT
> > > > >  call MatNestGetISs(A,isg,PETSC_NULL_OBJECT,ierr);CHKERRQ(ierr);
> > > > > print *, PETSC_NULL_OBJECT
> > > > >
> > > > >  call PetscFinalize(ierr)
> > > > >
> > > > > end program fieldsplittry2
> > > > > $ ./fieldsplittry2
> > > > >                     0
> > > > >              39367824
> > > > > $
> > > > >
> > > > >
> > > > > dr. ir. Christiaan Klaij
> > > > > CFD Researcher
> > > > > Research & Development
> > > > > E mailto:C.Klaij at marin.nl
> > > > > T +31 317 49 33 44
> > > > >
> > > > >
> > > > > MARIN
> > > > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
> > > > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl
> > > > >
> > >
> > >
> > > dr. ir. Christiaan Klaij
> > > CFD Researcher
> > > Research & Development
> > > E mailto:C.Klaij at marin.nl
> > > T +31 317 49 33 44
> > >
> > >
> > > MARIN
> > > 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
> > > T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl
> > >
> > >
> >
> 
> 


From vijay.gopal.c at gmail.com  Sat May  2 14:10:04 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Sat, 2 May 2015 21:10:04 +0200
Subject: [petsc-users] Matrix Assembly too slow
Message-ID: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>

Hi,

I'm trying to diagonalize large matrices using PETSc and SLEPc.
I've successfully diagonalized 100 million dimensional matrix in 2hours.

But, the problem is when i try to diagonailze (similar) a smaller metrix of
dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20
processors each (similar to the 100 million case), the matrix assembly
itself
takes 2h !

Could someone point out the mistakes that i'm making ?

i attach the source code and the output with the mail.

thanks,
 Vijay


#include <slepceps.h>
#include <petsctime.h>

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  Mat            A;           /* problem matrix */
  EPS            eps;         /* eigenproblem solver context */
  EPSType        type;
  PetscReal      error,tol,re,im;
  PetscScalar    kr,ki,value[52];
  Vec            xr,xi;
  PetscInt
n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
  PetscInt       d_nz,o_nz;
  PetscLogDouble t1,t2,tt1,tt2;
  PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
  PetscErrorCode ierr;
  PetscScalar    eigr;
  PetscScalar    eigi;
  PetscScalar    * data;
  Vec Vr,Vi;
  char           filename[PETSC_MAX_PATH_LEN]="FIL666";
  PetscViewer    viewer;
  PetscBool      ishermitian;
  int            mpiid;
  long int            kk,iii;
  long int            tcountcol,tcol[52];
  float          val[52];
  long int            ntrou=1;
  long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
  long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23,
22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
  long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
  double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
  double           xjjxy[52] ={0.0333333333333,-0.8,0.,
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
  double           xtt[52]   ={-1.0,0.,0.,
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

  SlepcInitialize(&argc,&argv,(char*)0,NULL);
  ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
n=%D\n\n",n);CHKERRQ(ierr);
  ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
//ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
  ierr =
MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
  ierr = MatSetUp(A);CHKERRQ(ierr);

  ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
  ierr =
MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
  MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
  ierr = PetscTime(&tt1);CHKERRQ(ierr);
  for (i=Istart; i<Iend; i++) {
      tcountcol=0;
      iii=i+1;
      if(i%5 == 0 && mpiid==0){
        ierr = PetscTime(&t1);CHKERRQ(ierr);
      }
      unit_l1_(
            l1,
            l2,
            ktyp,
            &iii,
            xjjxy,
            xjjz ,
            xtt  ,
            &tcountcol,
            &ntrou,
            tcol,
            val);
      for(kk=0;kk<52;kk++){
          value[kk] = val[kk];
          col[kk] = tcol[kk]-1;
      }
      if(i%5 == 0 && mpiid==0){
        ierr = PetscTime(&t2);CHKERRQ(ierr);
        ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime:
%f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
      }
      countcol=tcountcol;
      if(i%5 == 0 && mpiid==0){
        ierr = PetscTime(&t1);CHKERRQ(ierr);
      }
    ierr =
MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
      if(i%5 == 0 && mpiid==0){
        ierr = PetscTime(&t2);CHKERRQ(ierr);
        ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime:
%f\n",mpiid,t2-t1);CHKERRQ(ierr);
      }
  }
  ierr = PetscTime(&tt2);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix:
%f\n",tt2-tt1);CHKERRQ(ierr);


  ierr = PetscTime(&tt1);CHKERRQ(ierr);
  ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = PetscTime(&tt2);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix:
%f\n",tt2-tt1);CHKERRQ(ierr);
  ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
  ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);

  ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
  ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
  ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
  ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);

  ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
  tol = 1.e-8;
  maxit = 10000000;
  ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);

  ierr = PetscTime(&t1);CHKERRQ(ierr);
  ierr = EPSSolve(eps);CHKERRQ(ierr);
  ierr = PetscTime(&t2);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
%f\n",t2-t1);CHKERRQ(ierr);
  ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method:
%D\n",its);CHKERRQ(ierr);
  ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
%s\n\n",type);CHKERRQ(ierr);
  ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues:
%D\n",nev);CHKERRQ(ierr);
  ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g,
maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);

  ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
  ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
  /*
  EPSGetConverged(eps,&nconv);
  if (nconv>0) {
    PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
    EPSIsHermitian(eps,&ishermitian);
    for (i=0;i<nconv;i++) {
      EPSGetEigenvector(eps,i,xr,xi);
      VecView(xr,viewer);
#if !defined(PETSC_USE_COMPLEX)
      if (!ishermitian) { VecView(xi,viewer); }
#endif
    }
    PetscViewerDestroy(&viewer);
  }
  */
  ierr = EPSDestroy(&eps);CHKERRQ(ierr);
  ierr = MatDestroy(&A);CHKERRQ(ierr);
  ierr = VecDestroy(&xr);CHKERRQ(ierr);
  ierr = VecDestroy(&xi);CHKERRQ(ierr);
  ierr = SlepcFinalize();
  return 0;
}

OUTPUT:
 Time used to build the matrix: 1914.729022
[0] MatStashScatterBegin_Private(): No of messages: 0
[0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[6] MatStashScatterBegin_Private(): No of messages: 0
[6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[182] MatStashScatterBegin_Private(): No of messages: 0
[182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[100] MatStashScatterBegin_Private(): No of messages: 0
[100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[60] MatStashScatterBegin_Private(): No of messages: 0
[60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[140] MatStashScatterBegin_Private(): No of messages: 0
[140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[120] MatStashScatterBegin_Private(): No of messages: 0
[120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[162] MatStashScatterBegin_Private(): No of messages: 0
[162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[20] MatStashScatterBegin_Private(): No of messages: 0
[20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[80] MatStashScatterBegin_Private(): No of messages: 0
[80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[40] MatStashScatterBegin_Private(): No of messages: 0
[40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[7] MatStashScatterBegin_Private(): No of messages: 0
[7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[185] MatStashScatterBegin_Private(): No of messages: 0
[185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[101] MatStashScatterBegin_Private(): No of messages: 0
[101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[61] MatStashScatterBegin_Private(): No of messages: 0
[61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[142] MatStashScatterBegin_Private(): No of messages: 0
[142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[130] MatStashScatterBegin_Private(): No of messages: 0
[130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[166] MatStashScatterBegin_Private(): No of messages: 0
[166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[21] MatStashScatterBegin_Private(): No of messages: 0
[21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[81] MatStashScatterBegin_Private(): No of messages: 0
[81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[44] MatStashScatterBegin_Private(): No of messages: 0
[44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[9] MatStashScatterBegin_Private(): No of messages: 0
[9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[187] MatStashScatterBegin_Private(): No of messages: 0
[187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[105] MatStashScatterBegin_Private(): No of messages: 0
[105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[62] MatStashScatterBegin_Private(): No of messages: 0
[62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[143] MatStashScatterBegin_Private(): No of messages: 0
[143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[132] MatStashScatterBegin_Private(): No of messages: 0
[132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[169] MatStashScatterBegin_Private(): No of messages: 0
[169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[24] MatStashScatterBegin_Private(): No of messages: 0
[24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[84] MatStashScatterBegin_Private(): No of messages: 0
[84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[45] MatStashScatterBegin_Private(): No of messages: 0
[45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[10] MatStashScatterBegin_Private(): No of messages: 0
[10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[188] MatStashScatterBegin_Private(): No of messages: 0
[188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[109] MatStashScatterBegin_Private(): No of messages: 0
[109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[63] MatStashScatterBegin_Private(): No of messages: 0
[63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[148] MatStashScatterBegin_Private(): No of messages: 0
[148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[133] MatStashScatterBegin_Private(): No of messages: 0
[133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[170] MatStashScatterBegin_Private(): No of messages: 0
[170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[25] MatStashScatterBegin_Private(): No of messages: 0
[25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[86] MatStashScatterBegin_Private(): No of messages: 0
[86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[47] MatStashScatterBegin_Private(): No of messages: 0
[47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[11] MatStashScatterBegin_Private(): No of messages: 0
[11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[190] MatStashScatterBegin_Private(): No of messages: 0
[190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[111] MatStashScatterBegin_Private(): No of messages: 0
[111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[64] MatStashScatterBegin_Private(): No of messages: 0
[64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[156] MatStashScatterBegin_Private(): No of messages: 0
[156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[134] MatStashScatterBegin_Private(): No of messages: 0
[134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[171] MatStashScatterBegin_Private(): No of messages: 0
[171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[26] MatStashScatterBegin_Private(): No of messages: 0
[26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[91] MatStashScatterBegin_Private(): No of messages: 0
[91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[48] MatStashScatterBegin_Private(): No of messages: 0
[48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[12] MatStashScatterBegin_Private(): No of messages: 0
[12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[192] MatStashScatterBegin_Private(): No of messages: 0
[192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[112] MatStashScatterBegin_Private(): No of messages: 0
[112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[65] MatStashScatterBegin_Private(): No of messages: 0
[65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[157] MatStashScatterBegin_Private(): No of messages: 0
[157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[135] MatStashScatterBegin_Private(): No of messages: 0
[135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[174] MatStashScatterBegin_Private(): No of messages: 0
[174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[28] MatStashScatterBegin_Private(): No of messages: 0
[28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[13] MatStashScatterBegin_Private(): No of messages: 0
[13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[194] MatStashScatterBegin_Private(): No of messages: 0
[194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[113] MatStashScatterBegin_Private(): No of messages: 0
[113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[66] MatStashScatterBegin_Private(): No of messages: 0
[66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[158] MatStashScatterBegin_Private(): No of messages: 0
[158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[136] MatStashScatterBegin_Private(): No of messages: 0
[136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[160] MatStashScatterBegin_Private(): No of messages: 0
[160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[29] MatStashScatterBegin_Private(): No of messages: 0
[29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[92] MatStashScatterBegin_Private(): No of messages: 0
[92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[53] MatStashScatterBegin_Private(): No of messages: 0
[53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[14] MatStashScatterBegin_Private(): No of messages: 0
[14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[195] MatStashScatterBegin_Private(): No of messages: 0
[195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[114] MatStashScatterBegin_Private(): No of messages: 0
[114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[67] MatStashScatterBegin_Private(): No of messages: 0
[67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[141] MatStashScatterBegin_Private(): No of messages: 0
[141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[137] MatStashScatterBegin_Private(): No of messages: 0
[137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[161] MatStashScatterBegin_Private(): No of messages: 0
[161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[32] MatStashScatterBegin_Private(): No of messages: 0
[32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[93] MatStashScatterBegin_Private(): No of messages: 0
[93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[54] MatStashScatterBegin_Private(): No of messages: 0
[54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[15] MatStashScatterBegin_Private(): No of messages: 0
[15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[196] MatStashScatterBegin_Private(): No of messages: 0
[196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[115] MatStashScatterBegin_Private(): No of messages: 0
[115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[68] MatStashScatterBegin_Private(): No of messages: 0
[68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[144] MatStashScatterBegin_Private(): No of messages: 0
[144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[138] MatStashScatterBegin_Private(): No of messages: 0
[138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[163] MatStashScatterBegin_Private(): No of messages: 0
[163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[33] MatStashScatterBegin_Private(): No of messages: 0
[33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[94] MatStashScatterBegin_Private(): No of messages: 0
[94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[55] MatStashScatterBegin_Private(): No of messages: 0
[55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[17] MatStashScatterBegin_Private(): No of messages: 0
[17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[198] MatStashScatterBegin_Private(): No of messages: 0
[198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[116] MatStashScatterBegin_Private(): No of messages: 0
[116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[69] MatStashScatterBegin_Private(): No of messages: 0
[69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[145] MatStashScatterBegin_Private(): No of messages: 0
[145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[139] MatStashScatterBegin_Private(): No of messages: 0
[139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[164] MatStashScatterBegin_Private(): No of messages: 0
[164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[34] MatStashScatterBegin_Private(): No of messages: 0
[34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[95] MatStashScatterBegin_Private(): No of messages: 0
[95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[56] MatStashScatterBegin_Private(): No of messages: 0
[56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[1] MatStashScatterBegin_Private(): No of messages: 0
[1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[199] MatStashScatterBegin_Private(): No of messages: 0
[199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[119] MatStashScatterBegin_Private(): No of messages: 0
[119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[70] MatStashScatterBegin_Private(): No of messages: 0
[70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[146] MatStashScatterBegin_Private(): No of messages: 0
[146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[121] MatStashScatterBegin_Private(): No of messages: 0
[121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[165] MatStashScatterBegin_Private(): No of messages: 0
[165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[35] MatStashScatterBegin_Private(): No of messages: 0
[35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[99] MatStashScatterBegin_Private(): No of messages: 0
[99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[59] MatStashScatterBegin_Private(): No of messages: 0
[59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[2] MatStashScatterBegin_Private(): No of messages: 0
[2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[180] MatStashScatterBegin_Private(): No of messages: 0
[180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[102] MatStashScatterBegin_Private(): No of messages: 0
[102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[71] MatStashScatterBegin_Private(): No of messages: 0
[71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[147] MatStashScatterBegin_Private(): No of messages: 0
[147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[122] MatStashScatterBegin_Private(): No of messages: 0
[122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[167] MatStashScatterBegin_Private(): No of messages: 0
[167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[36] MatStashScatterBegin_Private(): No of messages: 0
[36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[82] MatStashScatterBegin_Private(): No of messages: 0
[82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[41] MatStashScatterBegin_Private(): No of messages: 0
[41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[3] MatStashScatterBegin_Private(): No of messages: 0
[3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[181] MatStashScatterBegin_Private(): No of messages: 0
[181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[103] MatStashScatterBegin_Private(): No of messages: 0
[103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[72] MatStashScatterBegin_Private(): No of messages: 0
[72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[149] MatStashScatterBegin_Private(): No of messages: 0
[149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[123] MatStashScatterBegin_Private(): No of messages: 0
[123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[168] MatStashScatterBegin_Private(): No of messages: 0
[168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[37] MatStashScatterBegin_Private(): No of messages: 0
[37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[83] MatStashScatterBegin_Private(): No of messages: 0
[83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[42] MatStashScatterBegin_Private(): No of messages: 0
[42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[4] MatStashScatterBegin_Private(): No of messages: 0
[4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[183] MatStashScatterBegin_Private(): No of messages: 0
[183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[104] MatStashScatterBegin_Private(): No of messages: 0
[104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[73] MatStashScatterBegin_Private(): No of messages: 0
[73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[150] MatStashScatterBegin_Private(): No of messages: 0
[150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[124] MatStashScatterBegin_Private(): No of messages: 0
[124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[172] MatStashScatterBegin_Private(): No of messages: 0
[172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[38] MatStashScatterBegin_Private(): No of messages: 0
[38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[85] MatStashScatterBegin_Private(): No of messages: 0
[85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[43] MatStashScatterBegin_Private(): No of messages: 0
[43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[5] MatStashScatterBegin_Private(): No of messages: 0
[5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[184] MatStashScatterBegin_Private(): No of messages: 0
[184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[106] MatStashScatterBegin_Private(): No of messages: 0
[106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[74] MatStashScatterBegin_Private(): No of messages: 0
[74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[151] MatStashScatterBegin_Private(): No of messages: 0
[151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[125] MatStashScatterBegin_Private(): No of messages: 0
[125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[173] MatStashScatterBegin_Private(): No of messages: 0
[173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[39] MatStashScatterBegin_Private(): No of messages: 0
[39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[87] MatStashScatterBegin_Private(): No of messages: 0
[87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[46] MatStashScatterBegin_Private(): No of messages: 0
[46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[8] MatStashScatterBegin_Private(): No of messages: 0
[8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[186] MatStashScatterBegin_Private(): No of messages: 0
[186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[107] MatStashScatterBegin_Private(): No of messages: 0
[107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[75] MatStashScatterBegin_Private(): No of messages: 0
[75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[152] MatStashScatterBegin_Private(): No of messages: 0
[152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[126] MatStashScatterBegin_Private(): No of messages: 0
[126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[175] MatStashScatterBegin_Private(): No of messages: 0
[175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[22] MatStashScatterBegin_Private(): No of messages: 0
[22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[88] MatStashScatterBegin_Private(): No of messages: 0
[88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[49] MatStashScatterBegin_Private(): No of messages: 0
[49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[16] MatStashScatterBegin_Private(): No of messages: 0
[16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[189] MatStashScatterBegin_Private(): No of messages: 0
[189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[108] MatStashScatterBegin_Private(): No of messages: 0
[108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[76] MatStashScatterBegin_Private(): No of messages: 0
[76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[153] MatStashScatterBegin_Private(): No of messages: 0
[153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[127] MatStashScatterBegin_Private(): No of messages: 0
[127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[176] MatStashScatterBegin_Private(): No of messages: 0
[176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[23] MatStashScatterBegin_Private(): No of messages: 0
[23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[89] MatStashScatterBegin_Private(): No of messages: 0
[89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[50] MatStashScatterBegin_Private(): No of messages: 0
[50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[18] MatStashScatterBegin_Private(): No of messages: 0
[18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[191] MatStashScatterBegin_Private(): No of messages: 0
[191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[110] MatStashScatterBegin_Private(): No of messages: 0
[110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[77] MatStashScatterBegin_Private(): No of messages: 0
[77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[154] MatStashScatterBegin_Private(): No of messages: 0
[154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[128] MatStashScatterBegin_Private(): No of messages: 0
[128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[177] MatStashScatterBegin_Private(): No of messages: 0
[177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[27] MatStashScatterBegin_Private(): No of messages: 0
[27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[90] MatStashScatterBegin_Private(): No of messages: 0
[90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[51] MatStashScatterBegin_Private(): No of messages: 0
[51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[19] MatStashScatterBegin_Private(): No of messages: 0
[19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[193] MatStashScatterBegin_Private(): No of messages: 0
[193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[117] MatStashScatterBegin_Private(): No of messages: 0
[117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[78] MatStashScatterBegin_Private(): No of messages: 0
[78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[155] MatStashScatterBegin_Private(): No of messages: 0
[155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[129] MatStashScatterBegin_Private(): No of messages: 0
[129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[178] MatStashScatterBegin_Private(): No of messages: 0
[178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[30] MatStashScatterBegin_Private(): No of messages: 0
[30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[96] MatStashScatterBegin_Private(): No of messages: 0
[96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[52] MatStashScatterBegin_Private(): No of messages: 0
[52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[197] MatStashScatterBegin_Private(): No of messages: 0
[197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[118] MatStashScatterBegin_Private(): No of messages: 0
[118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[79] MatStashScatterBegin_Private(): No of messages: 0
[79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[159] MatStashScatterBegin_Private(): No of messages: 0
[159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[131] MatStashScatterBegin_Private(): No of messages: 0
[131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[179] MatStashScatterBegin_Private(): No of messages: 0
[179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[31] MatStashScatterBegin_Private(): No of messages: 0
[31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[97] MatStashScatterBegin_Private(): No of messages: 0
[97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[57] MatStashScatterBegin_Private(): No of messages: 0
[57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[98] MatStashScatterBegin_Private(): No of messages: 0
[98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[58] MatStashScatterBegin_Private(): No of messages: 0
[58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3238253 unneeded,493497 used
[104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[104] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3197999 unneeded,533751 used
[4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[4] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3243809 unneeded,487941 used
[106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[106] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3243446 unneeded,488258 used
[144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
[144] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3237359 unneeded,494391 used
[123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[123] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3245082 unneeded,486622 used
[173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[173] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3217795 unneeded,513955 used
[85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[85] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3208633 unneeded,523117 used
[6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[6] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3209020 unneeded,522684 used
[185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[185] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3216305 unneeded,515445 used
[108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3201643 unneeded,530107 used
[73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[73] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3234040 unneeded,497664 used
[146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
[146] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3243435 unneeded,488315 used
[127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[127] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3249858 unneeded,481846 used
[177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[177] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3207727 unneeded,524023 used
[27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[27] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3220647 unneeded,511103 used
[87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
[87] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3197939 unneeded,533811 used
[54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[54] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3179441 unneeded,552309 used
[8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[8] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
3197890 unneeded,533814 used
[187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
[187] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
[108] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
[74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
3200381 unneeded,531369 used
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/d8856808/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex1.c
Type: text/x-csrc
Size: 6249 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/d8856808/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inp.log
Type: text/x-log
Size: 32349 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/d8856808/attachment-0001.bin>

From knepley at gmail.com  Sat May  2 14:15:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 2 May 2015 12:15:03 -0700
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
Message-ID: <CAMYG4GkNRuJ7+HAmCr9fVKD=0s8z=CkZBp47M7fkLF3Sxiycug@mail.gmail.com>

On Sat, May 2, 2015 at 12:10 PM, Vijay Gopal Chilkuri <
vijay.gopal.c at gmail.com> wrote:

> Hi,
>
> I'm trying to diagonalize large matrices using PETSc and SLEPc.
> I've successfully diagonalized 100 million dimensional matrix in 2hours.
>
> But, the problem is when i try to diagonailze (similar) a smaller metrix of
> dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20
> processors each (similar to the 100 million case), the matrix assembly
> itself
> takes 2h !
>
> Could someone point out the mistakes that i'm making ?
>

http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly

  Thanks,

     Matt


> i attach the source code and the output with the mail.
>
> thanks,
>  Vijay
>
>
> #include <slepceps.h>
> #include <petsctime.h>
>
> #undef __FUNCT__
> #define __FUNCT__ "main"
> int main(int argc,char **argv)
> {
>   Mat            A;           /* problem matrix */
>   EPS            eps;         /* eigenproblem solver context */
>   EPSType        type;
>   PetscReal      error,tol,re,im;
>   PetscScalar    kr,ki,value[52];
>   Vec            xr,xi;
>   PetscInt
> n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
>   PetscInt       d_nz,o_nz;
>   PetscLogDouble t1,t2,tt1,tt2;
>   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
>   PetscErrorCode ierr;
>   PetscScalar    eigr;
>   PetscScalar    eigi;
>   PetscScalar    * data;
>   Vec Vr,Vi;
>   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
>   PetscViewer    viewer;
>   PetscBool      ishermitian;
>   int            mpiid;
>   long int            kk,iii;
>   long int            tcountcol,tcol[52];
>   float          val[52];
>   long int            ntrou=1;
>   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3,
> 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23,
> 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
> 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
>   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>   double           xtt[52]   ={-1.0,0.,0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>
>   SlepcInitialize(&argc,&argv,(char*)0,NULL);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
> n=%D\n\n",n);CHKERRQ(ierr);
>   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
>   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>   ierr = MatSetUp(A);CHKERRQ(ierr);
>
>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>   for (i=Istart; i<Iend; i++) {
>       tcountcol=0;
>       iii=i+1;
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t1);CHKERRQ(ierr);
>       }
>       unit_l1_(
>             l1,
>             l2,
>             ktyp,
>             &iii,
>             xjjxy,
>             xjjz ,
>             xtt  ,
>             &tcountcol,
>             &ntrou,
>             tcol,
>             val);
>       for(kk=0;kk<52;kk++){
>           value[kk] = val[kk];
>           col[kk] = tcol[kk]-1;
>       }
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t2);CHKERRQ(ierr);
>         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime:
> %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
>       }
>       countcol=tcountcol;
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t1);CHKERRQ(ierr);
>       }
>     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t2);CHKERRQ(ierr);
>         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
>       }
>   }
>   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix:
> %f\n",tt2-tt1);CHKERRQ(ierr);
>
>
>   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix:
> %f\n",tt2-tt1);CHKERRQ(ierr);
>   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
>   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
>
>   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
>   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
>   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
>   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
>
>   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
>   tol = 1.e-8;
>   maxit = 10000000;
>   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
>
>   ierr = PetscTime(&t1);CHKERRQ(ierr);
>   ierr = EPSSolve(eps);CHKERRQ(ierr);
>   ierr = PetscTime(&t2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
> %f\n",t2-t1);CHKERRQ(ierr);
>   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the
> method: %D\n",its);CHKERRQ(ierr);
>   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
> %s\n\n",type);CHKERRQ(ierr);
>   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues:
> %D\n",nev);CHKERRQ(ierr);
>   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g,
> maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
>
>   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
>   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
>   /*
>   EPSGetConverged(eps,&nconv);
>   if (nconv>0) {
>     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
>     EPSIsHermitian(eps,&ishermitian);
>     for (i=0;i<nconv;i++) {
>       EPSGetEigenvector(eps,i,xr,xi);
>       VecView(xr,viewer);
> #if !defined(PETSC_USE_COMPLEX)
>       if (!ishermitian) { VecView(xi,viewer); }
> #endif
>     }
>     PetscViewerDestroy(&viewer);
>   }
>   */
>   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
>   ierr = MatDestroy(&A);CHKERRQ(ierr);
>   ierr = VecDestroy(&xr);CHKERRQ(ierr);
>   ierr = VecDestroy(&xi);CHKERRQ(ierr);
>   ierr = SlepcFinalize();
>   return 0;
> }
>
> OUTPUT:
>  Time used to build the matrix: 1914.729022
> [0] MatStashScatterBegin_Private(): No of messages: 0
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [6] MatStashScatterBegin_Private(): No of messages: 0
> [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [182] MatStashScatterBegin_Private(): No of messages: 0
> [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [100] MatStashScatterBegin_Private(): No of messages: 0
> [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [60] MatStashScatterBegin_Private(): No of messages: 0
> [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [140] MatStashScatterBegin_Private(): No of messages: 0
> [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [120] MatStashScatterBegin_Private(): No of messages: 0
> [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [162] MatStashScatterBegin_Private(): No of messages: 0
> [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [20] MatStashScatterBegin_Private(): No of messages: 0
> [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [80] MatStashScatterBegin_Private(): No of messages: 0
> [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [40] MatStashScatterBegin_Private(): No of messages: 0
> [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [7] MatStashScatterBegin_Private(): No of messages: 0
> [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [185] MatStashScatterBegin_Private(): No of messages: 0
> [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [101] MatStashScatterBegin_Private(): No of messages: 0
> [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [61] MatStashScatterBegin_Private(): No of messages: 0
> [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [142] MatStashScatterBegin_Private(): No of messages: 0
> [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [130] MatStashScatterBegin_Private(): No of messages: 0
> [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [166] MatStashScatterBegin_Private(): No of messages: 0
> [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [21] MatStashScatterBegin_Private(): No of messages: 0
> [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [81] MatStashScatterBegin_Private(): No of messages: 0
> [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [44] MatStashScatterBegin_Private(): No of messages: 0
> [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [9] MatStashScatterBegin_Private(): No of messages: 0
> [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [187] MatStashScatterBegin_Private(): No of messages: 0
> [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [105] MatStashScatterBegin_Private(): No of messages: 0
> [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [62] MatStashScatterBegin_Private(): No of messages: 0
> [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [143] MatStashScatterBegin_Private(): No of messages: 0
> [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [132] MatStashScatterBegin_Private(): No of messages: 0
> [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [169] MatStashScatterBegin_Private(): No of messages: 0
> [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [24] MatStashScatterBegin_Private(): No of messages: 0
> [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [84] MatStashScatterBegin_Private(): No of messages: 0
> [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [45] MatStashScatterBegin_Private(): No of messages: 0
> [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [10] MatStashScatterBegin_Private(): No of messages: 0
> [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [188] MatStashScatterBegin_Private(): No of messages: 0
> [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [109] MatStashScatterBegin_Private(): No of messages: 0
> [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [63] MatStashScatterBegin_Private(): No of messages: 0
> [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [148] MatStashScatterBegin_Private(): No of messages: 0
> [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [133] MatStashScatterBegin_Private(): No of messages: 0
> [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [170] MatStashScatterBegin_Private(): No of messages: 0
> [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [25] MatStashScatterBegin_Private(): No of messages: 0
> [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [86] MatStashScatterBegin_Private(): No of messages: 0
> [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [47] MatStashScatterBegin_Private(): No of messages: 0
> [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [11] MatStashScatterBegin_Private(): No of messages: 0
> [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [190] MatStashScatterBegin_Private(): No of messages: 0
> [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [111] MatStashScatterBegin_Private(): No of messages: 0
> [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [64] MatStashScatterBegin_Private(): No of messages: 0
> [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [156] MatStashScatterBegin_Private(): No of messages: 0
> [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [134] MatStashScatterBegin_Private(): No of messages: 0
> [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [171] MatStashScatterBegin_Private(): No of messages: 0
> [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [26] MatStashScatterBegin_Private(): No of messages: 0
> [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [91] MatStashScatterBegin_Private(): No of messages: 0
> [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [48] MatStashScatterBegin_Private(): No of messages: 0
> [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [12] MatStashScatterBegin_Private(): No of messages: 0
> [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [192] MatStashScatterBegin_Private(): No of messages: 0
> [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [112] MatStashScatterBegin_Private(): No of messages: 0
> [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [65] MatStashScatterBegin_Private(): No of messages: 0
> [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [157] MatStashScatterBegin_Private(): No of messages: 0
> [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [135] MatStashScatterBegin_Private(): No of messages: 0
> [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [174] MatStashScatterBegin_Private(): No of messages: 0
> [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [28] MatStashScatterBegin_Private(): No of messages: 0
> [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [13] MatStashScatterBegin_Private(): No of messages: 0
> [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [194] MatStashScatterBegin_Private(): No of messages: 0
> [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [113] MatStashScatterBegin_Private(): No of messages: 0
> [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [66] MatStashScatterBegin_Private(): No of messages: 0
> [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [158] MatStashScatterBegin_Private(): No of messages: 0
> [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [136] MatStashScatterBegin_Private(): No of messages: 0
> [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [160] MatStashScatterBegin_Private(): No of messages: 0
> [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [29] MatStashScatterBegin_Private(): No of messages: 0
> [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [92] MatStashScatterBegin_Private(): No of messages: 0
> [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [53] MatStashScatterBegin_Private(): No of messages: 0
> [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [14] MatStashScatterBegin_Private(): No of messages: 0
> [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [195] MatStashScatterBegin_Private(): No of messages: 0
> [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [114] MatStashScatterBegin_Private(): No of messages: 0
> [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [67] MatStashScatterBegin_Private(): No of messages: 0
> [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [141] MatStashScatterBegin_Private(): No of messages: 0
> [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [137] MatStashScatterBegin_Private(): No of messages: 0
> [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [161] MatStashScatterBegin_Private(): No of messages: 0
> [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [32] MatStashScatterBegin_Private(): No of messages: 0
> [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [93] MatStashScatterBegin_Private(): No of messages: 0
> [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [54] MatStashScatterBegin_Private(): No of messages: 0
> [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [15] MatStashScatterBegin_Private(): No of messages: 0
> [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [196] MatStashScatterBegin_Private(): No of messages: 0
> [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [115] MatStashScatterBegin_Private(): No of messages: 0
> [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [68] MatStashScatterBegin_Private(): No of messages: 0
> [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [144] MatStashScatterBegin_Private(): No of messages: 0
> [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [138] MatStashScatterBegin_Private(): No of messages: 0
> [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [163] MatStashScatterBegin_Private(): No of messages: 0
> [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [33] MatStashScatterBegin_Private(): No of messages: 0
> [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [94] MatStashScatterBegin_Private(): No of messages: 0
> [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [55] MatStashScatterBegin_Private(): No of messages: 0
> [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [17] MatStashScatterBegin_Private(): No of messages: 0
> [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [198] MatStashScatterBegin_Private(): No of messages: 0
> [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [116] MatStashScatterBegin_Private(): No of messages: 0
> [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [69] MatStashScatterBegin_Private(): No of messages: 0
> [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [145] MatStashScatterBegin_Private(): No of messages: 0
> [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [139] MatStashScatterBegin_Private(): No of messages: 0
> [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [164] MatStashScatterBegin_Private(): No of messages: 0
> [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [34] MatStashScatterBegin_Private(): No of messages: 0
> [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [95] MatStashScatterBegin_Private(): No of messages: 0
> [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [56] MatStashScatterBegin_Private(): No of messages: 0
> [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [1] MatStashScatterBegin_Private(): No of messages: 0
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [199] MatStashScatterBegin_Private(): No of messages: 0
> [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [119] MatStashScatterBegin_Private(): No of messages: 0
> [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [70] MatStashScatterBegin_Private(): No of messages: 0
> [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [146] MatStashScatterBegin_Private(): No of messages: 0
> [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [121] MatStashScatterBegin_Private(): No of messages: 0
> [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [165] MatStashScatterBegin_Private(): No of messages: 0
> [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [35] MatStashScatterBegin_Private(): No of messages: 0
> [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [99] MatStashScatterBegin_Private(): No of messages: 0
> [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [59] MatStashScatterBegin_Private(): No of messages: 0
> [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [2] MatStashScatterBegin_Private(): No of messages: 0
> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [180] MatStashScatterBegin_Private(): No of messages: 0
> [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [102] MatStashScatterBegin_Private(): No of messages: 0
> [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [71] MatStashScatterBegin_Private(): No of messages: 0
> [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [147] MatStashScatterBegin_Private(): No of messages: 0
> [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [122] MatStashScatterBegin_Private(): No of messages: 0
> [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [167] MatStashScatterBegin_Private(): No of messages: 0
> [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [36] MatStashScatterBegin_Private(): No of messages: 0
> [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [82] MatStashScatterBegin_Private(): No of messages: 0
> [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [41] MatStashScatterBegin_Private(): No of messages: 0
> [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [3] MatStashScatterBegin_Private(): No of messages: 0
> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [181] MatStashScatterBegin_Private(): No of messages: 0
> [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [103] MatStashScatterBegin_Private(): No of messages: 0
> [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [72] MatStashScatterBegin_Private(): No of messages: 0
> [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [149] MatStashScatterBegin_Private(): No of messages: 0
> [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [123] MatStashScatterBegin_Private(): No of messages: 0
> [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [168] MatStashScatterBegin_Private(): No of messages: 0
> [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [37] MatStashScatterBegin_Private(): No of messages: 0
> [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [83] MatStashScatterBegin_Private(): No of messages: 0
> [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [42] MatStashScatterBegin_Private(): No of messages: 0
> [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [4] MatStashScatterBegin_Private(): No of messages: 0
> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [183] MatStashScatterBegin_Private(): No of messages: 0
> [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [104] MatStashScatterBegin_Private(): No of messages: 0
> [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [73] MatStashScatterBegin_Private(): No of messages: 0
> [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [150] MatStashScatterBegin_Private(): No of messages: 0
> [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [124] MatStashScatterBegin_Private(): No of messages: 0
> [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [172] MatStashScatterBegin_Private(): No of messages: 0
> [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [38] MatStashScatterBegin_Private(): No of messages: 0
> [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [85] MatStashScatterBegin_Private(): No of messages: 0
> [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [43] MatStashScatterBegin_Private(): No of messages: 0
> [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [5] MatStashScatterBegin_Private(): No of messages: 0
> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [184] MatStashScatterBegin_Private(): No of messages: 0
> [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [106] MatStashScatterBegin_Private(): No of messages: 0
> [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [74] MatStashScatterBegin_Private(): No of messages: 0
> [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [151] MatStashScatterBegin_Private(): No of messages: 0
> [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [125] MatStashScatterBegin_Private(): No of messages: 0
> [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [173] MatStashScatterBegin_Private(): No of messages: 0
> [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [39] MatStashScatterBegin_Private(): No of messages: 0
> [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [87] MatStashScatterBegin_Private(): No of messages: 0
> [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [46] MatStashScatterBegin_Private(): No of messages: 0
> [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [8] MatStashScatterBegin_Private(): No of messages: 0
> [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [186] MatStashScatterBegin_Private(): No of messages: 0
> [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [107] MatStashScatterBegin_Private(): No of messages: 0
> [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [75] MatStashScatterBegin_Private(): No of messages: 0
> [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [152] MatStashScatterBegin_Private(): No of messages: 0
> [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [126] MatStashScatterBegin_Private(): No of messages: 0
> [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [175] MatStashScatterBegin_Private(): No of messages: 0
> [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [22] MatStashScatterBegin_Private(): No of messages: 0
> [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [88] MatStashScatterBegin_Private(): No of messages: 0
> [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [49] MatStashScatterBegin_Private(): No of messages: 0
> [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [16] MatStashScatterBegin_Private(): No of messages: 0
> [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [189] MatStashScatterBegin_Private(): No of messages: 0
> [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [108] MatStashScatterBegin_Private(): No of messages: 0
> [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [76] MatStashScatterBegin_Private(): No of messages: 0
> [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [153] MatStashScatterBegin_Private(): No of messages: 0
> [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [127] MatStashScatterBegin_Private(): No of messages: 0
> [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [176] MatStashScatterBegin_Private(): No of messages: 0
> [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [23] MatStashScatterBegin_Private(): No of messages: 0
> [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [89] MatStashScatterBegin_Private(): No of messages: 0
> [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [50] MatStashScatterBegin_Private(): No of messages: 0
> [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [18] MatStashScatterBegin_Private(): No of messages: 0
> [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [191] MatStashScatterBegin_Private(): No of messages: 0
> [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [110] MatStashScatterBegin_Private(): No of messages: 0
> [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [77] MatStashScatterBegin_Private(): No of messages: 0
> [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [154] MatStashScatterBegin_Private(): No of messages: 0
> [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [128] MatStashScatterBegin_Private(): No of messages: 0
> [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [177] MatStashScatterBegin_Private(): No of messages: 0
> [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [27] MatStashScatterBegin_Private(): No of messages: 0
> [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [90] MatStashScatterBegin_Private(): No of messages: 0
> [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [51] MatStashScatterBegin_Private(): No of messages: 0
> [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [19] MatStashScatterBegin_Private(): No of messages: 0
> [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [193] MatStashScatterBegin_Private(): No of messages: 0
> [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [117] MatStashScatterBegin_Private(): No of messages: 0
> [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [78] MatStashScatterBegin_Private(): No of messages: 0
> [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [155] MatStashScatterBegin_Private(): No of messages: 0
> [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [129] MatStashScatterBegin_Private(): No of messages: 0
> [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [178] MatStashScatterBegin_Private(): No of messages: 0
> [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [30] MatStashScatterBegin_Private(): No of messages: 0
> [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [96] MatStashScatterBegin_Private(): No of messages: 0
> [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [52] MatStashScatterBegin_Private(): No of messages: 0
> [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [197] MatStashScatterBegin_Private(): No of messages: 0
> [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [118] MatStashScatterBegin_Private(): No of messages: 0
> [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [79] MatStashScatterBegin_Private(): No of messages: 0
> [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [159] MatStashScatterBegin_Private(): No of messages: 0
> [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [131] MatStashScatterBegin_Private(): No of messages: 0
> [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [179] MatStashScatterBegin_Private(): No of messages: 0
> [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [31] MatStashScatterBegin_Private(): No of messages: 0
> [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [97] MatStashScatterBegin_Private(): No of messages: 0
> [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [57] MatStashScatterBegin_Private(): No of messages: 0
> [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [98] MatStashScatterBegin_Private(): No of messages: 0
> [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [58] MatStashScatterBegin_Private(): No of messages: 0
> [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3238253 unneeded,493497 used
> [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3197999 unneeded,533751 used
> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3243809 unneeded,487941 used
> [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3243446 unneeded,488258 used
> [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3237359 unneeded,494391 used
> [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3245082 unneeded,486622 used
> [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3217795 unneeded,513955 used
> [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3208633 unneeded,523117 used
> [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3209020 unneeded,522684 used
> [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3216305 unneeded,515445 used
> [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3201643 unneeded,530107 used
> [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3234040 unneeded,497664 used
> [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3243435 unneeded,488315 used
> [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3249858 unneeded,481846 used
> [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3207727 unneeded,524023 used
> [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3220647 unneeded,511103 used
> [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3197939 unneeded,533811 used
> [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3179441 unneeded,552309 used
> [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space:
> 3197890 unneeded,533814 used
> [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3200381 unneeded,531369 used
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/5df1e10f/attachment-0001.html>

From bsmith at mcs.anl.gov  Sat May  2 14:42:50 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 2 May 2015 14:42:50 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
Message-ID: <9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>


  You have two hundred processes banging on the same file on the same filesystem fighting with each other to read in the matrix. That's like hiring two hundred guys to dig a ditch but making them share one shovel; you won't get speed up, you'll get speed down.


  Make a simple sequential PETSc program that reads in the matrix and then store it with MatView() in binary format. Then in your parallel program use MatLoad() to read the matrix in (it will take a few seconds to read in the matrix in that case).

  Barry


> On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> Hi,
> 
> I'm trying to diagonalize large matrices using PETSc and SLEPc.
> I've successfully diagonalized 100 million dimensional matrix in 2hours.
> 
> But, the problem is when i try to diagonailze (similar) a smaller metrix of
> dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20 processors each (similar to the 100 million case), the matrix assembly itself
> takes 2h ! 
> 
> Could someone point out the mistakes that i'm making ?
> 
> i attach the source code and the output with the mail.
> 
> thanks,
>  Vijay
> 
> 
> #include <slepceps.h>
> #include <petsctime.h>
> 
> #undef __FUNCT__
> #define __FUNCT__ "main"
> int main(int argc,char **argv)
> {
>   Mat            A;           /* problem matrix */
>   EPS            eps;         /* eigenproblem solver context */
>   EPSType        type;
>   PetscReal      error,tol,re,im;
>   PetscScalar    kr,ki,value[52];
>   Vec            xr,xi;
>   PetscInt       n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
>   PetscInt       d_nz,o_nz;
>   PetscLogDouble t1,t2,tt1,tt2;
>   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
>   PetscErrorCode ierr;
>   PetscScalar    eigr;
>   PetscScalar    eigi;
>   PetscScalar    * data;
>   Vec		 Vr,Vi;
>   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
>   PetscViewer    viewer;
>   PetscBool      ishermitian;
>   int            mpiid;
>   long int            kk,iii;
>   long int            tcountcol,tcol[52];
>   float          val[52];
>   long int            ntrou=1;
>   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
>   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; 
>   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; 
>   double           xtt[52]   ={-1.0,0.,0.,
>       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}; 
> 
>   SlepcInitialize(&argc,&argv,(char*)0,NULL);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem, n=%D\n\n",n);CHKERRQ(ierr);
>   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
>   ierr = MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>   ierr = MatSetUp(A);CHKERRQ(ierr);
> 
>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>   ierr = MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>   for (i=Istart; i<Iend; i++) {
>       tcountcol=0;
>       iii=i+1;
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t1);CHKERRQ(ierr);
>       }
>       unit_l1_(
>             l1,
>             l2,
>             ktyp,
>             &iii,
>             xjjxy,
>             xjjz ,
>             xtt  ,
>             &tcountcol,
>             &ntrou,
>             tcol,
>             val);
>       for(kk=0;kk<52;kk++){
>           value[kk] = val[kk];
>           col[kk] = tcol[kk]-1;
>       }
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t2);CHKERRQ(ierr);
>         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
>       }
>       countcol=tcountcol;
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t1);CHKERRQ(ierr);
>       }
>     ierr = MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>       if(i%5 == 0 && mpiid==0){
>         ierr = PetscTime(&t2);CHKERRQ(ierr);
>         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
>       }
>   }
>   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> 
> 
>   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
>   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> 
>   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
>   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
>   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
>   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> 
>   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
>   tol = 1.e-8;
>   maxit = 10000000;
>   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> 
>   ierr = PetscTime(&t1);CHKERRQ(ierr);
>   ierr = EPSSolve(eps);CHKERRQ(ierr);
>   ierr = PetscTime(&t2);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used: %f\n",t2-t1);CHKERRQ(ierr);
>   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method: %D\n",its);CHKERRQ(ierr);
>   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method: %s\n\n",type);CHKERRQ(ierr);
>   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
>   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> 
>   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
>   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
>   /*
>   EPSGetConverged(eps,&nconv);
>   if (nconv>0) {
>     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
>     EPSIsHermitian(eps,&ishermitian);
>     for (i=0;i<nconv;i++) {
>       EPSGetEigenvector(eps,i,xr,xi);
>       VecView(xr,viewer);
> #if !defined(PETSC_USE_COMPLEX)
>       if (!ishermitian) { VecView(xi,viewer); }
> #endif
>     }
>     PetscViewerDestroy(&viewer);
>   }
>   */
>   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
>   ierr = MatDestroy(&A);CHKERRQ(ierr);
>   ierr = VecDestroy(&xr);CHKERRQ(ierr);
>   ierr = VecDestroy(&xi);CHKERRQ(ierr);
>   ierr = SlepcFinalize();
>   return 0;
> }
> 
> OUTPUT:
>  Time used to build the matrix: 1914.729022
> [0] MatStashScatterBegin_Private(): No of messages: 0 
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [6] MatStashScatterBegin_Private(): No of messages: 0 
> [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [182] MatStashScatterBegin_Private(): No of messages: 0 
> [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [100] MatStashScatterBegin_Private(): No of messages: 0 
> [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [60] MatStashScatterBegin_Private(): No of messages: 0 
> [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [140] MatStashScatterBegin_Private(): No of messages: 0 
> [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [120] MatStashScatterBegin_Private(): No of messages: 0 
> [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [162] MatStashScatterBegin_Private(): No of messages: 0 
> [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [20] MatStashScatterBegin_Private(): No of messages: 0 
> [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [80] MatStashScatterBegin_Private(): No of messages: 0 
> [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [40] MatStashScatterBegin_Private(): No of messages: 0 
> [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [7] MatStashScatterBegin_Private(): No of messages: 0 
> [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [185] MatStashScatterBegin_Private(): No of messages: 0 
> [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [101] MatStashScatterBegin_Private(): No of messages: 0 
> [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [61] MatStashScatterBegin_Private(): No of messages: 0 
> [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [142] MatStashScatterBegin_Private(): No of messages: 0 
> [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [130] MatStashScatterBegin_Private(): No of messages: 0 
> [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [166] MatStashScatterBegin_Private(): No of messages: 0 
> [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [21] MatStashScatterBegin_Private(): No of messages: 0 
> [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [81] MatStashScatterBegin_Private(): No of messages: 0 
> [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [44] MatStashScatterBegin_Private(): No of messages: 0 
> [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [9] MatStashScatterBegin_Private(): No of messages: 0 
> [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [187] MatStashScatterBegin_Private(): No of messages: 0 
> [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [105] MatStashScatterBegin_Private(): No of messages: 0 
> [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [62] MatStashScatterBegin_Private(): No of messages: 0 
> [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [143] MatStashScatterBegin_Private(): No of messages: 0 
> [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [132] MatStashScatterBegin_Private(): No of messages: 0 
> [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [169] MatStashScatterBegin_Private(): No of messages: 0 
> [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [24] MatStashScatterBegin_Private(): No of messages: 0 
> [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [84] MatStashScatterBegin_Private(): No of messages: 0 
> [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [45] MatStashScatterBegin_Private(): No of messages: 0 
> [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [10] MatStashScatterBegin_Private(): No of messages: 0 
> [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [188] MatStashScatterBegin_Private(): No of messages: 0 
> [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [109] MatStashScatterBegin_Private(): No of messages: 0 
> [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [63] MatStashScatterBegin_Private(): No of messages: 0 
> [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [148] MatStashScatterBegin_Private(): No of messages: 0 
> [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [133] MatStashScatterBegin_Private(): No of messages: 0 
> [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [170] MatStashScatterBegin_Private(): No of messages: 0 
> [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [25] MatStashScatterBegin_Private(): No of messages: 0 
> [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [86] MatStashScatterBegin_Private(): No of messages: 0 
> [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [47] MatStashScatterBegin_Private(): No of messages: 0 
> [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [11] MatStashScatterBegin_Private(): No of messages: 0 
> [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [190] MatStashScatterBegin_Private(): No of messages: 0 
> [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [111] MatStashScatterBegin_Private(): No of messages: 0 
> [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [64] MatStashScatterBegin_Private(): No of messages: 0 
> [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [156] MatStashScatterBegin_Private(): No of messages: 0 
> [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [134] MatStashScatterBegin_Private(): No of messages: 0 
> [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [171] MatStashScatterBegin_Private(): No of messages: 0 
> [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [26] MatStashScatterBegin_Private(): No of messages: 0 
> [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [91] MatStashScatterBegin_Private(): No of messages: 0 
> [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [48] MatStashScatterBegin_Private(): No of messages: 0 
> [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [12] MatStashScatterBegin_Private(): No of messages: 0 
> [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [192] MatStashScatterBegin_Private(): No of messages: 0 
> [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [112] MatStashScatterBegin_Private(): No of messages: 0 
> [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [65] MatStashScatterBegin_Private(): No of messages: 0 
> [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [157] MatStashScatterBegin_Private(): No of messages: 0 
> [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [135] MatStashScatterBegin_Private(): No of messages: 0 
> [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [174] MatStashScatterBegin_Private(): No of messages: 0 
> [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [28] MatStashScatterBegin_Private(): No of messages: 0 
> [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [13] MatStashScatterBegin_Private(): No of messages: 0 
> [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [194] MatStashScatterBegin_Private(): No of messages: 0 
> [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [113] MatStashScatterBegin_Private(): No of messages: 0 
> [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [66] MatStashScatterBegin_Private(): No of messages: 0 
> [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [158] MatStashScatterBegin_Private(): No of messages: 0 
> [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [136] MatStashScatterBegin_Private(): No of messages: 0 
> [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [160] MatStashScatterBegin_Private(): No of messages: 0 
> [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [29] MatStashScatterBegin_Private(): No of messages: 0 
> [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [92] MatStashScatterBegin_Private(): No of messages: 0 
> [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [53] MatStashScatterBegin_Private(): No of messages: 0 
> [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [14] MatStashScatterBegin_Private(): No of messages: 0 
> [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [195] MatStashScatterBegin_Private(): No of messages: 0 
> [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [114] MatStashScatterBegin_Private(): No of messages: 0 
> [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [67] MatStashScatterBegin_Private(): No of messages: 0 
> [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [141] MatStashScatterBegin_Private(): No of messages: 0 
> [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [137] MatStashScatterBegin_Private(): No of messages: 0 
> [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [161] MatStashScatterBegin_Private(): No of messages: 0 
> [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [32] MatStashScatterBegin_Private(): No of messages: 0 
> [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [93] MatStashScatterBegin_Private(): No of messages: 0 
> [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [54] MatStashScatterBegin_Private(): No of messages: 0 
> [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [15] MatStashScatterBegin_Private(): No of messages: 0 
> [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [196] MatStashScatterBegin_Private(): No of messages: 0 
> [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [115] MatStashScatterBegin_Private(): No of messages: 0 
> [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [68] MatStashScatterBegin_Private(): No of messages: 0 
> [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [144] MatStashScatterBegin_Private(): No of messages: 0 
> [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [138] MatStashScatterBegin_Private(): No of messages: 0 
> [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [163] MatStashScatterBegin_Private(): No of messages: 0 
> [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [33] MatStashScatterBegin_Private(): No of messages: 0 
> [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [94] MatStashScatterBegin_Private(): No of messages: 0 
> [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [55] MatStashScatterBegin_Private(): No of messages: 0 
> [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [17] MatStashScatterBegin_Private(): No of messages: 0 
> [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [198] MatStashScatterBegin_Private(): No of messages: 0 
> [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [116] MatStashScatterBegin_Private(): No of messages: 0 
> [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [69] MatStashScatterBegin_Private(): No of messages: 0 
> [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [145] MatStashScatterBegin_Private(): No of messages: 0 
> [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [139] MatStashScatterBegin_Private(): No of messages: 0 
> [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [164] MatStashScatterBegin_Private(): No of messages: 0 
> [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [34] MatStashScatterBegin_Private(): No of messages: 0 
> [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [95] MatStashScatterBegin_Private(): No of messages: 0 
> [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [56] MatStashScatterBegin_Private(): No of messages: 0 
> [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [1] MatStashScatterBegin_Private(): No of messages: 0 
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [199] MatStashScatterBegin_Private(): No of messages: 0 
> [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [119] MatStashScatterBegin_Private(): No of messages: 0 
> [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [70] MatStashScatterBegin_Private(): No of messages: 0 
> [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [146] MatStashScatterBegin_Private(): No of messages: 0 
> [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [121] MatStashScatterBegin_Private(): No of messages: 0 
> [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [165] MatStashScatterBegin_Private(): No of messages: 0 
> [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [35] MatStashScatterBegin_Private(): No of messages: 0 
> [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [99] MatStashScatterBegin_Private(): No of messages: 0 
> [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [59] MatStashScatterBegin_Private(): No of messages: 0 
> [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [2] MatStashScatterBegin_Private(): No of messages: 0 
> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [180] MatStashScatterBegin_Private(): No of messages: 0 
> [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [102] MatStashScatterBegin_Private(): No of messages: 0 
> [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [71] MatStashScatterBegin_Private(): No of messages: 0 
> [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [147] MatStashScatterBegin_Private(): No of messages: 0 
> [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [122] MatStashScatterBegin_Private(): No of messages: 0 
> [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [167] MatStashScatterBegin_Private(): No of messages: 0 
> [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [36] MatStashScatterBegin_Private(): No of messages: 0 
> [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [82] MatStashScatterBegin_Private(): No of messages: 0 
> [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [41] MatStashScatterBegin_Private(): No of messages: 0 
> [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [3] MatStashScatterBegin_Private(): No of messages: 0 
> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [181] MatStashScatterBegin_Private(): No of messages: 0 
> [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [103] MatStashScatterBegin_Private(): No of messages: 0 
> [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [72] MatStashScatterBegin_Private(): No of messages: 0 
> [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [149] MatStashScatterBegin_Private(): No of messages: 0 
> [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [123] MatStashScatterBegin_Private(): No of messages: 0 
> [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [168] MatStashScatterBegin_Private(): No of messages: 0 
> [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [37] MatStashScatterBegin_Private(): No of messages: 0 
> [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [83] MatStashScatterBegin_Private(): No of messages: 0 
> [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [42] MatStashScatterBegin_Private(): No of messages: 0 
> [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [4] MatStashScatterBegin_Private(): No of messages: 0 
> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [183] MatStashScatterBegin_Private(): No of messages: 0 
> [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [104] MatStashScatterBegin_Private(): No of messages: 0 
> [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [73] MatStashScatterBegin_Private(): No of messages: 0 
> [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [150] MatStashScatterBegin_Private(): No of messages: 0 
> [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [124] MatStashScatterBegin_Private(): No of messages: 0 
> [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [172] MatStashScatterBegin_Private(): No of messages: 0 
> [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [38] MatStashScatterBegin_Private(): No of messages: 0 
> [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [85] MatStashScatterBegin_Private(): No of messages: 0 
> [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [43] MatStashScatterBegin_Private(): No of messages: 0 
> [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [5] MatStashScatterBegin_Private(): No of messages: 0 
> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [184] MatStashScatterBegin_Private(): No of messages: 0 
> [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [106] MatStashScatterBegin_Private(): No of messages: 0 
> [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [74] MatStashScatterBegin_Private(): No of messages: 0 
> [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [151] MatStashScatterBegin_Private(): No of messages: 0 
> [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [125] MatStashScatterBegin_Private(): No of messages: 0 
> [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [173] MatStashScatterBegin_Private(): No of messages: 0 
> [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [39] MatStashScatterBegin_Private(): No of messages: 0 
> [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [87] MatStashScatterBegin_Private(): No of messages: 0 
> [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [46] MatStashScatterBegin_Private(): No of messages: 0 
> [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [8] MatStashScatterBegin_Private(): No of messages: 0 
> [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [186] MatStashScatterBegin_Private(): No of messages: 0 
> [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [107] MatStashScatterBegin_Private(): No of messages: 0 
> [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [75] MatStashScatterBegin_Private(): No of messages: 0 
> [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [152] MatStashScatterBegin_Private(): No of messages: 0 
> [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [126] MatStashScatterBegin_Private(): No of messages: 0 
> [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [175] MatStashScatterBegin_Private(): No of messages: 0 
> [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [22] MatStashScatterBegin_Private(): No of messages: 0 
> [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [88] MatStashScatterBegin_Private(): No of messages: 0 
> [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [49] MatStashScatterBegin_Private(): No of messages: 0 
> [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [16] MatStashScatterBegin_Private(): No of messages: 0 
> [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [189] MatStashScatterBegin_Private(): No of messages: 0 
> [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [108] MatStashScatterBegin_Private(): No of messages: 0 
> [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [76] MatStashScatterBegin_Private(): No of messages: 0 
> [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [153] MatStashScatterBegin_Private(): No of messages: 0 
> [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [127] MatStashScatterBegin_Private(): No of messages: 0 
> [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [176] MatStashScatterBegin_Private(): No of messages: 0 
> [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [23] MatStashScatterBegin_Private(): No of messages: 0 
> [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [89] MatStashScatterBegin_Private(): No of messages: 0 
> [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [50] MatStashScatterBegin_Private(): No of messages: 0 
> [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [18] MatStashScatterBegin_Private(): No of messages: 0 
> [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [191] MatStashScatterBegin_Private(): No of messages: 0 
> [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [110] MatStashScatterBegin_Private(): No of messages: 0 
> [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [77] MatStashScatterBegin_Private(): No of messages: 0 
> [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [154] MatStashScatterBegin_Private(): No of messages: 0 
> [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [128] MatStashScatterBegin_Private(): No of messages: 0 
> [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [177] MatStashScatterBegin_Private(): No of messages: 0 
> [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [27] MatStashScatterBegin_Private(): No of messages: 0 
> [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [90] MatStashScatterBegin_Private(): No of messages: 0 
> [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [51] MatStashScatterBegin_Private(): No of messages: 0 
> [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [19] MatStashScatterBegin_Private(): No of messages: 0 
> [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [193] MatStashScatterBegin_Private(): No of messages: 0 
> [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [117] MatStashScatterBegin_Private(): No of messages: 0 
> [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [78] MatStashScatterBegin_Private(): No of messages: 0 
> [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [155] MatStashScatterBegin_Private(): No of messages: 0 
> [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [129] MatStashScatterBegin_Private(): No of messages: 0 
> [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [178] MatStashScatterBegin_Private(): No of messages: 0 
> [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [30] MatStashScatterBegin_Private(): No of messages: 0 
> [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [96] MatStashScatterBegin_Private(): No of messages: 0 
> [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [52] MatStashScatterBegin_Private(): No of messages: 0 
> [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [197] MatStashScatterBegin_Private(): No of messages: 0 
> [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [118] MatStashScatterBegin_Private(): No of messages: 0 
> [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [79] MatStashScatterBegin_Private(): No of messages: 0 
> [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [159] MatStashScatterBegin_Private(): No of messages: 0 
> [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [131] MatStashScatterBegin_Private(): No of messages: 0 
> [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [179] MatStashScatterBegin_Private(): No of messages: 0 
> [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [31] MatStashScatterBegin_Private(): No of messages: 0 
> [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [97] MatStashScatterBegin_Private(): No of messages: 0 
> [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [57] MatStashScatterBegin_Private(): No of messages: 0 
> [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [98] MatStashScatterBegin_Private(): No of messages: 0 
> [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [58] MatStashScatterBegin_Private(): No of messages: 0 
> [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3238253 unneeded,493497 used
> [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [104] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197999 unneeded,533751 used
> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243809 unneeded,487941 used
> [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [106] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3243446 unneeded,488258 used
> [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> [144] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3237359 unneeded,494391 used
> [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [123] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3245082 unneeded,486622 used
> [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [173] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3217795 unneeded,513955 used
> [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [85] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3208633 unneeded,523117 used
> [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [6] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3209020 unneeded,522684 used
> [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [185] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3216305 unneeded,515445 used
> [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3201643 unneeded,530107 used
> [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [73] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3234040 unneeded,497664 used
> [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> [146] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243435 unneeded,488315 used
> [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [127] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3249858 unneeded,481846 used
> [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [177] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3207727 unneeded,524023 used
> [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [27] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3220647 unneeded,511103 used
> [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> [87] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197939 unneeded,533811 used
> [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [54] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3179441 unneeded,552309 used
> [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [8] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3197890 unneeded,533814 used
> [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> [187] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> [108] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3200381 unneeded,531369 used
> 
> <ex1.c><inp.log>


From vijay.gopal.c at gmail.com  Sat May  2 14:57:21 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Sat, 2 May 2015 21:57:21 +0200
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
Message-ID: <CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>

Please let me clarify,

@Matt:
I've followed the proper manner to rapid assembly.
That is :
  1. preallocation
  2. nonzero ownership
Despite of having done the proper preallocation (as you can see from the
output file),
the matrix takes ages to assemble, whereas the same code works for a larger
matrix with (100,000,000) elements taking 2 min to assemble !

@Barry:
I am not touching any file in the code that I attachd.
What i'm doing is that each processor will (according to it's ownership
range) get it's nonzero entries
using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their
respective rows.
Again to be clear I"m NOT doing file I/O anywhere.

Could you please have a look at the files attached.





On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   You have two hundred processes banging on the same file on the same
> filesystem fighting with each other to read in the matrix. That's like
> hiring two hundred guys to dig a ditch but making them share one shovel;
> you won't get speed up, you'll get speed down.
>
>
>   Make a simple sequential PETSc program that reads in the matrix and then
> store it with MatView() in binary format. Then in your parallel program use
> MatLoad() to read the matrix in (it will take a few seconds to read in the
> matrix in that case).
>
>   Barry
>
>
> > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> >
> > Hi,
> >
> > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > I've successfully diagonalized 100 million dimensional matrix in 2hours.
> >
> > But, the problem is when i try to diagonailze (similar) a smaller metrix
> of
> > dimension 67,000,000 (24 nonzero elements per row) with 10 noes running
> 20 processors each (similar to the 100 million case), the matrix assembly
> itself
> > takes 2h !
> >
> > Could someone point out the mistakes that i'm making ?
> >
> > i attach the source code and the output with the mail.
> >
> > thanks,
> >  Vijay
> >
> >
> > #include <slepceps.h>
> > #include <petsctime.h>
> >
> > #undef __FUNCT__
> > #define __FUNCT__ "main"
> > int main(int argc,char **argv)
> > {
> >   Mat            A;           /* problem matrix */
> >   EPS            eps;         /* eigenproblem solver context */
> >   EPSType        type;
> >   PetscReal      error,tol,re,im;
> >   PetscScalar    kr,ki,value[52];
> >   Vec            xr,xi;
> >   PetscInt
>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> >   PetscInt       d_nz,o_nz;
> >   PetscLogDouble t1,t2,tt1,tt2;
> >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> >   PetscErrorCode ierr;
> >   PetscScalar    eigr;
> >   PetscScalar    eigi;
> >   PetscScalar    * data;
> >   Vec          Vr,Vi;
> >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> >   PetscViewer    viewer;
> >   PetscBool      ishermitian;
> >   int            mpiid;
> >   long int            kk,iii;
> >   long int            tcountcol,tcol[52];
> >   float          val[52];
> >   long int            ntrou=1;
> >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2,
> 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
> 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24,
> 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21,
> 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >   double           xtt[52]   ={-1.0,0.,0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >
> >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
> n=%D\n\n",n);CHKERRQ(ierr);
> >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> >   ierr = MatSetUp(A);CHKERRQ(ierr);
> >
> >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> >   for (i=Istart; i<Iend; i++) {
> >       tcountcol=0;
> >       iii=i+1;
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> >       }
> >       unit_l1_(
> >             l1,
> >             l2,
> >             ktyp,
> >             &iii,
> >             xjjxy,
> >             xjjz ,
> >             xtt  ,
> >             &tcountcol,
> >             &ntrou,
> >             tcol,
> >             val);
> >       for(kk=0;kk<52;kk++){
> >           value[kk] = val[kk];
> >           col[kk] = tcol[kk]-1;
> >       }
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime:
> %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> >       }
> >       countcol=tcountcol;
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> >       }
> >     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> >       }
> >   }
> >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix:
> %f\n",tt2-tt1);CHKERRQ(ierr);
> >
> >
> >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> >
> >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> >
> >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> >   tol = 1.e-8;
> >   maxit = 10000000;
> >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> >
> >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
> %f\n",t2-t1);CHKERRQ(ierr);
> >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the
> method: %D\n",its);CHKERRQ(ierr);
> >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
> %s\n\n",type);CHKERRQ(ierr);
> >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues:
> %D\n",nev);CHKERRQ(ierr);
> >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g,
> maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> >
> >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> >   /*
> >   EPSGetConverged(eps,&nconv);
> >   if (nconv>0) {
> >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> >     EPSIsHermitian(eps,&ishermitian);
> >     for (i=0;i<nconv;i++) {
> >       EPSGetEigenvector(eps,i,xr,xi);
> >       VecView(xr,viewer);
> > #if !defined(PETSC_USE_COMPLEX)
> >       if (!ishermitian) { VecView(xi,viewer); }
> > #endif
> >     }
> >     PetscViewerDestroy(&viewer);
> >   }
> >   */
> >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> >   ierr = SlepcFinalize();
> >   return 0;
> > }
> >
> > OUTPUT:
> >  Time used to build the matrix: 1914.729022
> > [0] MatStashScatterBegin_Private(): No of messages: 0
> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [6] MatStashScatterBegin_Private(): No of messages: 0
> > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [182] MatStashScatterBegin_Private(): No of messages: 0
> > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [100] MatStashScatterBegin_Private(): No of messages: 0
> > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [60] MatStashScatterBegin_Private(): No of messages: 0
> > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [140] MatStashScatterBegin_Private(): No of messages: 0
> > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [120] MatStashScatterBegin_Private(): No of messages: 0
> > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [162] MatStashScatterBegin_Private(): No of messages: 0
> > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [20] MatStashScatterBegin_Private(): No of messages: 0
> > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [80] MatStashScatterBegin_Private(): No of messages: 0
> > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [40] MatStashScatterBegin_Private(): No of messages: 0
> > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [7] MatStashScatterBegin_Private(): No of messages: 0
> > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [185] MatStashScatterBegin_Private(): No of messages: 0
> > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [101] MatStashScatterBegin_Private(): No of messages: 0
> > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [61] MatStashScatterBegin_Private(): No of messages: 0
> > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [142] MatStashScatterBegin_Private(): No of messages: 0
> > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [130] MatStashScatterBegin_Private(): No of messages: 0
> > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [166] MatStashScatterBegin_Private(): No of messages: 0
> > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [21] MatStashScatterBegin_Private(): No of messages: 0
> > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [81] MatStashScatterBegin_Private(): No of messages: 0
> > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [44] MatStashScatterBegin_Private(): No of messages: 0
> > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [9] MatStashScatterBegin_Private(): No of messages: 0
> > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [187] MatStashScatterBegin_Private(): No of messages: 0
> > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [105] MatStashScatterBegin_Private(): No of messages: 0
> > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [62] MatStashScatterBegin_Private(): No of messages: 0
> > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [143] MatStashScatterBegin_Private(): No of messages: 0
> > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [132] MatStashScatterBegin_Private(): No of messages: 0
> > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [169] MatStashScatterBegin_Private(): No of messages: 0
> > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [24] MatStashScatterBegin_Private(): No of messages: 0
> > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [84] MatStashScatterBegin_Private(): No of messages: 0
> > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [45] MatStashScatterBegin_Private(): No of messages: 0
> > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [10] MatStashScatterBegin_Private(): No of messages: 0
> > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [188] MatStashScatterBegin_Private(): No of messages: 0
> > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [109] MatStashScatterBegin_Private(): No of messages: 0
> > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [63] MatStashScatterBegin_Private(): No of messages: 0
> > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [148] MatStashScatterBegin_Private(): No of messages: 0
> > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [133] MatStashScatterBegin_Private(): No of messages: 0
> > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [170] MatStashScatterBegin_Private(): No of messages: 0
> > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [25] MatStashScatterBegin_Private(): No of messages: 0
> > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [86] MatStashScatterBegin_Private(): No of messages: 0
> > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [47] MatStashScatterBegin_Private(): No of messages: 0
> > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [11] MatStashScatterBegin_Private(): No of messages: 0
> > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [190] MatStashScatterBegin_Private(): No of messages: 0
> > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [111] MatStashScatterBegin_Private(): No of messages: 0
> > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [64] MatStashScatterBegin_Private(): No of messages: 0
> > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [156] MatStashScatterBegin_Private(): No of messages: 0
> > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [134] MatStashScatterBegin_Private(): No of messages: 0
> > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [171] MatStashScatterBegin_Private(): No of messages: 0
> > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [26] MatStashScatterBegin_Private(): No of messages: 0
> > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [91] MatStashScatterBegin_Private(): No of messages: 0
> > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [48] MatStashScatterBegin_Private(): No of messages: 0
> > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [12] MatStashScatterBegin_Private(): No of messages: 0
> > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [192] MatStashScatterBegin_Private(): No of messages: 0
> > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [112] MatStashScatterBegin_Private(): No of messages: 0
> > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [65] MatStashScatterBegin_Private(): No of messages: 0
> > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [157] MatStashScatterBegin_Private(): No of messages: 0
> > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [135] MatStashScatterBegin_Private(): No of messages: 0
> > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [174] MatStashScatterBegin_Private(): No of messages: 0
> > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [28] MatStashScatterBegin_Private(): No of messages: 0
> > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [13] MatStashScatterBegin_Private(): No of messages: 0
> > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [194] MatStashScatterBegin_Private(): No of messages: 0
> > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [113] MatStashScatterBegin_Private(): No of messages: 0
> > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [66] MatStashScatterBegin_Private(): No of messages: 0
> > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [158] MatStashScatterBegin_Private(): No of messages: 0
> > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [136] MatStashScatterBegin_Private(): No of messages: 0
> > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [160] MatStashScatterBegin_Private(): No of messages: 0
> > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [29] MatStashScatterBegin_Private(): No of messages: 0
> > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [92] MatStashScatterBegin_Private(): No of messages: 0
> > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [53] MatStashScatterBegin_Private(): No of messages: 0
> > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [14] MatStashScatterBegin_Private(): No of messages: 0
> > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [195] MatStashScatterBegin_Private(): No of messages: 0
> > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [114] MatStashScatterBegin_Private(): No of messages: 0
> > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [67] MatStashScatterBegin_Private(): No of messages: 0
> > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [141] MatStashScatterBegin_Private(): No of messages: 0
> > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [137] MatStashScatterBegin_Private(): No of messages: 0
> > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [161] MatStashScatterBegin_Private(): No of messages: 0
> > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [32] MatStashScatterBegin_Private(): No of messages: 0
> > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [93] MatStashScatterBegin_Private(): No of messages: 0
> > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [54] MatStashScatterBegin_Private(): No of messages: 0
> > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [15] MatStashScatterBegin_Private(): No of messages: 0
> > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [196] MatStashScatterBegin_Private(): No of messages: 0
> > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [115] MatStashScatterBegin_Private(): No of messages: 0
> > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [68] MatStashScatterBegin_Private(): No of messages: 0
> > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [144] MatStashScatterBegin_Private(): No of messages: 0
> > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [138] MatStashScatterBegin_Private(): No of messages: 0
> > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [163] MatStashScatterBegin_Private(): No of messages: 0
> > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [33] MatStashScatterBegin_Private(): No of messages: 0
> > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [94] MatStashScatterBegin_Private(): No of messages: 0
> > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [55] MatStashScatterBegin_Private(): No of messages: 0
> > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [17] MatStashScatterBegin_Private(): No of messages: 0
> > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [198] MatStashScatterBegin_Private(): No of messages: 0
> > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [116] MatStashScatterBegin_Private(): No of messages: 0
> > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [69] MatStashScatterBegin_Private(): No of messages: 0
> > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [145] MatStashScatterBegin_Private(): No of messages: 0
> > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [139] MatStashScatterBegin_Private(): No of messages: 0
> > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [164] MatStashScatterBegin_Private(): No of messages: 0
> > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [34] MatStashScatterBegin_Private(): No of messages: 0
> > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [95] MatStashScatterBegin_Private(): No of messages: 0
> > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [56] MatStashScatterBegin_Private(): No of messages: 0
> > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [1] MatStashScatterBegin_Private(): No of messages: 0
> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [199] MatStashScatterBegin_Private(): No of messages: 0
> > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [119] MatStashScatterBegin_Private(): No of messages: 0
> > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [70] MatStashScatterBegin_Private(): No of messages: 0
> > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [146] MatStashScatterBegin_Private(): No of messages: 0
> > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [121] MatStashScatterBegin_Private(): No of messages: 0
> > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [165] MatStashScatterBegin_Private(): No of messages: 0
> > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [35] MatStashScatterBegin_Private(): No of messages: 0
> > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [99] MatStashScatterBegin_Private(): No of messages: 0
> > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [59] MatStashScatterBegin_Private(): No of messages: 0
> > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [2] MatStashScatterBegin_Private(): No of messages: 0
> > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [180] MatStashScatterBegin_Private(): No of messages: 0
> > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [102] MatStashScatterBegin_Private(): No of messages: 0
> > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [71] MatStashScatterBegin_Private(): No of messages: 0
> > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [147] MatStashScatterBegin_Private(): No of messages: 0
> > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [122] MatStashScatterBegin_Private(): No of messages: 0
> > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [167] MatStashScatterBegin_Private(): No of messages: 0
> > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [36] MatStashScatterBegin_Private(): No of messages: 0
> > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [82] MatStashScatterBegin_Private(): No of messages: 0
> > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [41] MatStashScatterBegin_Private(): No of messages: 0
> > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [3] MatStashScatterBegin_Private(): No of messages: 0
> > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [181] MatStashScatterBegin_Private(): No of messages: 0
> > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [103] MatStashScatterBegin_Private(): No of messages: 0
> > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [72] MatStashScatterBegin_Private(): No of messages: 0
> > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [149] MatStashScatterBegin_Private(): No of messages: 0
> > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [123] MatStashScatterBegin_Private(): No of messages: 0
> > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [168] MatStashScatterBegin_Private(): No of messages: 0
> > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [37] MatStashScatterBegin_Private(): No of messages: 0
> > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [83] MatStashScatterBegin_Private(): No of messages: 0
> > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [42] MatStashScatterBegin_Private(): No of messages: 0
> > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [4] MatStashScatterBegin_Private(): No of messages: 0
> > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [183] MatStashScatterBegin_Private(): No of messages: 0
> > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [104] MatStashScatterBegin_Private(): No of messages: 0
> > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [73] MatStashScatterBegin_Private(): No of messages: 0
> > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [150] MatStashScatterBegin_Private(): No of messages: 0
> > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [124] MatStashScatterBegin_Private(): No of messages: 0
> > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [172] MatStashScatterBegin_Private(): No of messages: 0
> > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [38] MatStashScatterBegin_Private(): No of messages: 0
> > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [85] MatStashScatterBegin_Private(): No of messages: 0
> > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [43] MatStashScatterBegin_Private(): No of messages: 0
> > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [5] MatStashScatterBegin_Private(): No of messages: 0
> > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [184] MatStashScatterBegin_Private(): No of messages: 0
> > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [106] MatStashScatterBegin_Private(): No of messages: 0
> > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [74] MatStashScatterBegin_Private(): No of messages: 0
> > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [151] MatStashScatterBegin_Private(): No of messages: 0
> > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [125] MatStashScatterBegin_Private(): No of messages: 0
> > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [173] MatStashScatterBegin_Private(): No of messages: 0
> > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [39] MatStashScatterBegin_Private(): No of messages: 0
> > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [87] MatStashScatterBegin_Private(): No of messages: 0
> > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [46] MatStashScatterBegin_Private(): No of messages: 0
> > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [8] MatStashScatterBegin_Private(): No of messages: 0
> > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [186] MatStashScatterBegin_Private(): No of messages: 0
> > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [107] MatStashScatterBegin_Private(): No of messages: 0
> > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [75] MatStashScatterBegin_Private(): No of messages: 0
> > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [152] MatStashScatterBegin_Private(): No of messages: 0
> > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [126] MatStashScatterBegin_Private(): No of messages: 0
> > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [175] MatStashScatterBegin_Private(): No of messages: 0
> > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [22] MatStashScatterBegin_Private(): No of messages: 0
> > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [88] MatStashScatterBegin_Private(): No of messages: 0
> > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [49] MatStashScatterBegin_Private(): No of messages: 0
> > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [16] MatStashScatterBegin_Private(): No of messages: 0
> > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [189] MatStashScatterBegin_Private(): No of messages: 0
> > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [108] MatStashScatterBegin_Private(): No of messages: 0
> > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [76] MatStashScatterBegin_Private(): No of messages: 0
> > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [153] MatStashScatterBegin_Private(): No of messages: 0
> > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [127] MatStashScatterBegin_Private(): No of messages: 0
> > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [176] MatStashScatterBegin_Private(): No of messages: 0
> > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [23] MatStashScatterBegin_Private(): No of messages: 0
> > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [89] MatStashScatterBegin_Private(): No of messages: 0
> > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [50] MatStashScatterBegin_Private(): No of messages: 0
> > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [18] MatStashScatterBegin_Private(): No of messages: 0
> > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [191] MatStashScatterBegin_Private(): No of messages: 0
> > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [110] MatStashScatterBegin_Private(): No of messages: 0
> > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [77] MatStashScatterBegin_Private(): No of messages: 0
> > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [154] MatStashScatterBegin_Private(): No of messages: 0
> > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [128] MatStashScatterBegin_Private(): No of messages: 0
> > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [177] MatStashScatterBegin_Private(): No of messages: 0
> > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [27] MatStashScatterBegin_Private(): No of messages: 0
> > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [90] MatStashScatterBegin_Private(): No of messages: 0
> > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [51] MatStashScatterBegin_Private(): No of messages: 0
> > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [19] MatStashScatterBegin_Private(): No of messages: 0
> > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [193] MatStashScatterBegin_Private(): No of messages: 0
> > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [117] MatStashScatterBegin_Private(): No of messages: 0
> > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [78] MatStashScatterBegin_Private(): No of messages: 0
> > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [155] MatStashScatterBegin_Private(): No of messages: 0
> > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [129] MatStashScatterBegin_Private(): No of messages: 0
> > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [178] MatStashScatterBegin_Private(): No of messages: 0
> > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [30] MatStashScatterBegin_Private(): No of messages: 0
> > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [96] MatStashScatterBegin_Private(): No of messages: 0
> > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [52] MatStashScatterBegin_Private(): No of messages: 0
> > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [197] MatStashScatterBegin_Private(): No of messages: 0
> > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [118] MatStashScatterBegin_Private(): No of messages: 0
> > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [79] MatStashScatterBegin_Private(): No of messages: 0
> > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [159] MatStashScatterBegin_Private(): No of messages: 0
> > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [131] MatStashScatterBegin_Private(): No of messages: 0
> > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [179] MatStashScatterBegin_Private(): No of messages: 0
> > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [31] MatStashScatterBegin_Private(): No of messages: 0
> > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [97] MatStashScatterBegin_Private(): No of messages: 0
> > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [57] MatStashScatterBegin_Private(): No of messages: 0
> > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [98] MatStashScatterBegin_Private(): No of messages: 0
> > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [58] MatStashScatterBegin_Private(): No of messages: 0
> > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3238253 unneeded,493497 used
> > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3197999 unneeded,533751 used
> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243809 unneeded,487941 used
> > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3243446 unneeded,488258 used
> > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3237359 unneeded,494391 used
> > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3245082 unneeded,486622 used
> > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3217795 unneeded,513955 used
> > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3208633 unneeded,523117 used
> > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3209020 unneeded,522684 used
> > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3216305 unneeded,515445 used
> > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3201643 unneeded,530107 used
> > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3234040 unneeded,497664 used
> > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243435 unneeded,488315 used
> > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3249858 unneeded,481846 used
> > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3207727 unneeded,524023 used
> > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3220647 unneeded,511103 used
> > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3197939 unneeded,533811 used
> > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is
> 0
> > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3179441 unneeded,552309 used
> > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3197890 unneeded,533814 used
> > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space:
> 3200381 unneeded,531369 used
> >
> > <ex1.c><inp.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/271d1e74/attachment-0001.html>

From bsmith at mcs.anl.gov  Sat May  2 15:26:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 2 May 2015 15:26:44 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
Message-ID: <C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>


> On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> Please let me clarify,
> 
> @Matt: 
> I've followed the proper manner to rapid assembly.
> That is :
>   1. preallocation
>   2. nonzero ownership
> Despite of having done the proper preallocation (as you can see from the output file),
> the matrix takes ages to assemble, whereas the same code works for a larger matrix with (100,000,000) elements taking 2 min to assemble !
> 
> @Barry:
> I am not touching any file in the code that I attachd.
> What i'm doing is that each processor will (according to it's ownership range) get it's nonzero entries
> using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their respective rows.
> Again to be clear I"m NOT doing file I/O anywhere.

  Oh sorry, where is the fortran code? I assumed the fortran code was reading from a file.

   Based on what you sent yes it should take little time; if you send something I could run I could try to see why it would be slow.

  Barry

> 
> Could you please have a look at the files attached.
> 
> 
> 
> 
> 
> On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   You have two hundred processes banging on the same file on the same filesystem fighting with each other to read in the matrix. That's like hiring two hundred guys to dig a ditch but making them share one shovel; you won't get speed up, you'll get speed down.
> 
> 
>   Make a simple sequential PETSc program that reads in the matrix and then store it with MatView() in binary format. Then in your parallel program use MatLoad() to read the matrix in (it will take a few seconds to read in the matrix in that case).
> 
>   Barry
> 
> 
> > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> >
> > Hi,
> >
> > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > I've successfully diagonalized 100 million dimensional matrix in 2hours.
> >
> > But, the problem is when i try to diagonailze (similar) a smaller metrix of
> > dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20 processors each (similar to the 100 million case), the matrix assembly itself
> > takes 2h !
> >
> > Could someone point out the mistakes that i'm making ?
> >
> > i attach the source code and the output with the mail.
> >
> > thanks,
> >  Vijay
> >
> >
> > #include <slepceps.h>
> > #include <petsctime.h>
> >
> > #undef __FUNCT__
> > #define __FUNCT__ "main"
> > int main(int argc,char **argv)
> > {
> >   Mat            A;           /* problem matrix */
> >   EPS            eps;         /* eigenproblem solver context */
> >   EPSType        type;
> >   PetscReal      error,tol,re,im;
> >   PetscScalar    kr,ki,value[52];
> >   Vec            xr,xi;
> >   PetscInt       n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> >   PetscInt       d_nz,o_nz;
> >   PetscLogDouble t1,t2,tt1,tt2;
> >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> >   PetscErrorCode ierr;
> >   PetscScalar    eigr;
> >   PetscScalar    eigi;
> >   PetscScalar    * data;
> >   Vec          Vr,Vi;
> >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> >   PetscViewer    viewer;
> >   PetscBool      ishermitian;
> >   int            mpiid;
> >   long int            kk,iii;
> >   long int            tcountcol,tcol[52];
> >   float          val[52];
> >   long int            ntrou=1;
> >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >   double           xtt[52]   ={-1.0,0.,0.,
> >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> >
> >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem, n=%D\n\n",n);CHKERRQ(ierr);
> >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> >   ierr = MatSetUp(A);CHKERRQ(ierr);
> >
> >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> >   for (i=Istart; i<Iend; i++) {
> >       tcountcol=0;
> >       iii=i+1;
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> >       }
> >       unit_l1_(
> >             l1,
> >             l2,
> >             ktyp,
> >             &iii,
> >             xjjxy,
> >             xjjz ,
> >             xtt  ,
> >             &tcountcol,
> >             &ntrou,
> >             tcol,
> >             val);
> >       for(kk=0;kk<52;kk++){
> >           value[kk] = val[kk];
> >           col[kk] = tcol[kk]-1;
> >       }
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> >       }
> >       countcol=tcountcol;
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> >       }
> >     ierr = MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> >       if(i%5 == 0 && mpiid==0){
> >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> >       }
> >   }
> >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> >
> >
> >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> >
> >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> >
> >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> >   tol = 1.e-8;
> >   maxit = 10000000;
> >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> >
> >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used: %f\n",t2-t1);CHKERRQ(ierr);
> >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method: %D\n",its);CHKERRQ(ierr);
> >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method: %s\n\n",type);CHKERRQ(ierr);
> >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
> >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> >
> >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> >   /*
> >   EPSGetConverged(eps,&nconv);
> >   if (nconv>0) {
> >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> >     EPSIsHermitian(eps,&ishermitian);
> >     for (i=0;i<nconv;i++) {
> >       EPSGetEigenvector(eps,i,xr,xi);
> >       VecView(xr,viewer);
> > #if !defined(PETSC_USE_COMPLEX)
> >       if (!ishermitian) { VecView(xi,viewer); }
> > #endif
> >     }
> >     PetscViewerDestroy(&viewer);
> >   }
> >   */
> >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> >   ierr = SlepcFinalize();
> >   return 0;
> > }
> >
> > OUTPUT:
> >  Time used to build the matrix: 1914.729022
> > [0] MatStashScatterBegin_Private(): No of messages: 0
> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [6] MatStashScatterBegin_Private(): No of messages: 0
> > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [182] MatStashScatterBegin_Private(): No of messages: 0
> > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [100] MatStashScatterBegin_Private(): No of messages: 0
> > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [60] MatStashScatterBegin_Private(): No of messages: 0
> > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [140] MatStashScatterBegin_Private(): No of messages: 0
> > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [120] MatStashScatterBegin_Private(): No of messages: 0
> > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [162] MatStashScatterBegin_Private(): No of messages: 0
> > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [20] MatStashScatterBegin_Private(): No of messages: 0
> > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [80] MatStashScatterBegin_Private(): No of messages: 0
> > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [40] MatStashScatterBegin_Private(): No of messages: 0
> > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [7] MatStashScatterBegin_Private(): No of messages: 0
> > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [185] MatStashScatterBegin_Private(): No of messages: 0
> > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [101] MatStashScatterBegin_Private(): No of messages: 0
> > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [61] MatStashScatterBegin_Private(): No of messages: 0
> > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [142] MatStashScatterBegin_Private(): No of messages: 0
> > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [130] MatStashScatterBegin_Private(): No of messages: 0
> > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [166] MatStashScatterBegin_Private(): No of messages: 0
> > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [21] MatStashScatterBegin_Private(): No of messages: 0
> > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [81] MatStashScatterBegin_Private(): No of messages: 0
> > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [44] MatStashScatterBegin_Private(): No of messages: 0
> > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [9] MatStashScatterBegin_Private(): No of messages: 0
> > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [187] MatStashScatterBegin_Private(): No of messages: 0
> > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [105] MatStashScatterBegin_Private(): No of messages: 0
> > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [62] MatStashScatterBegin_Private(): No of messages: 0
> > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [143] MatStashScatterBegin_Private(): No of messages: 0
> > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [132] MatStashScatterBegin_Private(): No of messages: 0
> > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [169] MatStashScatterBegin_Private(): No of messages: 0
> > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [24] MatStashScatterBegin_Private(): No of messages: 0
> > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [84] MatStashScatterBegin_Private(): No of messages: 0
> > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [45] MatStashScatterBegin_Private(): No of messages: 0
> > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [10] MatStashScatterBegin_Private(): No of messages: 0
> > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [188] MatStashScatterBegin_Private(): No of messages: 0
> > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [109] MatStashScatterBegin_Private(): No of messages: 0
> > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [63] MatStashScatterBegin_Private(): No of messages: 0
> > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [148] MatStashScatterBegin_Private(): No of messages: 0
> > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [133] MatStashScatterBegin_Private(): No of messages: 0
> > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [170] MatStashScatterBegin_Private(): No of messages: 0
> > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [25] MatStashScatterBegin_Private(): No of messages: 0
> > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [86] MatStashScatterBegin_Private(): No of messages: 0
> > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [47] MatStashScatterBegin_Private(): No of messages: 0
> > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [11] MatStashScatterBegin_Private(): No of messages: 0
> > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [190] MatStashScatterBegin_Private(): No of messages: 0
> > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [111] MatStashScatterBegin_Private(): No of messages: 0
> > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [64] MatStashScatterBegin_Private(): No of messages: 0
> > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [156] MatStashScatterBegin_Private(): No of messages: 0
> > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [134] MatStashScatterBegin_Private(): No of messages: 0
> > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [171] MatStashScatterBegin_Private(): No of messages: 0
> > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [26] MatStashScatterBegin_Private(): No of messages: 0
> > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [91] MatStashScatterBegin_Private(): No of messages: 0
> > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [48] MatStashScatterBegin_Private(): No of messages: 0
> > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [12] MatStashScatterBegin_Private(): No of messages: 0
> > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [192] MatStashScatterBegin_Private(): No of messages: 0
> > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [112] MatStashScatterBegin_Private(): No of messages: 0
> > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [65] MatStashScatterBegin_Private(): No of messages: 0
> > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [157] MatStashScatterBegin_Private(): No of messages: 0
> > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [135] MatStashScatterBegin_Private(): No of messages: 0
> > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [174] MatStashScatterBegin_Private(): No of messages: 0
> > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [28] MatStashScatterBegin_Private(): No of messages: 0
> > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [13] MatStashScatterBegin_Private(): No of messages: 0
> > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [194] MatStashScatterBegin_Private(): No of messages: 0
> > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [113] MatStashScatterBegin_Private(): No of messages: 0
> > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [66] MatStashScatterBegin_Private(): No of messages: 0
> > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [158] MatStashScatterBegin_Private(): No of messages: 0
> > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [136] MatStashScatterBegin_Private(): No of messages: 0
> > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [160] MatStashScatterBegin_Private(): No of messages: 0
> > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [29] MatStashScatterBegin_Private(): No of messages: 0
> > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [92] MatStashScatterBegin_Private(): No of messages: 0
> > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [53] MatStashScatterBegin_Private(): No of messages: 0
> > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [14] MatStashScatterBegin_Private(): No of messages: 0
> > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [195] MatStashScatterBegin_Private(): No of messages: 0
> > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [114] MatStashScatterBegin_Private(): No of messages: 0
> > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [67] MatStashScatterBegin_Private(): No of messages: 0
> > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [141] MatStashScatterBegin_Private(): No of messages: 0
> > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [137] MatStashScatterBegin_Private(): No of messages: 0
> > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [161] MatStashScatterBegin_Private(): No of messages: 0
> > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [32] MatStashScatterBegin_Private(): No of messages: 0
> > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [93] MatStashScatterBegin_Private(): No of messages: 0
> > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [54] MatStashScatterBegin_Private(): No of messages: 0
> > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [15] MatStashScatterBegin_Private(): No of messages: 0
> > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [196] MatStashScatterBegin_Private(): No of messages: 0
> > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [115] MatStashScatterBegin_Private(): No of messages: 0
> > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [68] MatStashScatterBegin_Private(): No of messages: 0
> > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [144] MatStashScatterBegin_Private(): No of messages: 0
> > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [138] MatStashScatterBegin_Private(): No of messages: 0
> > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [163] MatStashScatterBegin_Private(): No of messages: 0
> > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [33] MatStashScatterBegin_Private(): No of messages: 0
> > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [94] MatStashScatterBegin_Private(): No of messages: 0
> > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [55] MatStashScatterBegin_Private(): No of messages: 0
> > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [17] MatStashScatterBegin_Private(): No of messages: 0
> > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [198] MatStashScatterBegin_Private(): No of messages: 0
> > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [116] MatStashScatterBegin_Private(): No of messages: 0
> > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [69] MatStashScatterBegin_Private(): No of messages: 0
> > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [145] MatStashScatterBegin_Private(): No of messages: 0
> > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [139] MatStashScatterBegin_Private(): No of messages: 0
> > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [164] MatStashScatterBegin_Private(): No of messages: 0
> > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [34] MatStashScatterBegin_Private(): No of messages: 0
> > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [95] MatStashScatterBegin_Private(): No of messages: 0
> > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [56] MatStashScatterBegin_Private(): No of messages: 0
> > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [1] MatStashScatterBegin_Private(): No of messages: 0
> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [199] MatStashScatterBegin_Private(): No of messages: 0
> > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [119] MatStashScatterBegin_Private(): No of messages: 0
> > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [70] MatStashScatterBegin_Private(): No of messages: 0
> > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [146] MatStashScatterBegin_Private(): No of messages: 0
> > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [121] MatStashScatterBegin_Private(): No of messages: 0
> > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [165] MatStashScatterBegin_Private(): No of messages: 0
> > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [35] MatStashScatterBegin_Private(): No of messages: 0
> > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [99] MatStashScatterBegin_Private(): No of messages: 0
> > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [59] MatStashScatterBegin_Private(): No of messages: 0
> > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [2] MatStashScatterBegin_Private(): No of messages: 0
> > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [180] MatStashScatterBegin_Private(): No of messages: 0
> > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [102] MatStashScatterBegin_Private(): No of messages: 0
> > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [71] MatStashScatterBegin_Private(): No of messages: 0
> > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [147] MatStashScatterBegin_Private(): No of messages: 0
> > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [122] MatStashScatterBegin_Private(): No of messages: 0
> > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [167] MatStashScatterBegin_Private(): No of messages: 0
> > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [36] MatStashScatterBegin_Private(): No of messages: 0
> > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [82] MatStashScatterBegin_Private(): No of messages: 0
> > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [41] MatStashScatterBegin_Private(): No of messages: 0
> > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [3] MatStashScatterBegin_Private(): No of messages: 0
> > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [181] MatStashScatterBegin_Private(): No of messages: 0
> > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [103] MatStashScatterBegin_Private(): No of messages: 0
> > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [72] MatStashScatterBegin_Private(): No of messages: 0
> > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [149] MatStashScatterBegin_Private(): No of messages: 0
> > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [123] MatStashScatterBegin_Private(): No of messages: 0
> > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [168] MatStashScatterBegin_Private(): No of messages: 0
> > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [37] MatStashScatterBegin_Private(): No of messages: 0
> > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [83] MatStashScatterBegin_Private(): No of messages: 0
> > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [42] MatStashScatterBegin_Private(): No of messages: 0
> > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [4] MatStashScatterBegin_Private(): No of messages: 0
> > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [183] MatStashScatterBegin_Private(): No of messages: 0
> > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [104] MatStashScatterBegin_Private(): No of messages: 0
> > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [73] MatStashScatterBegin_Private(): No of messages: 0
> > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [150] MatStashScatterBegin_Private(): No of messages: 0
> > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [124] MatStashScatterBegin_Private(): No of messages: 0
> > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [172] MatStashScatterBegin_Private(): No of messages: 0
> > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [38] MatStashScatterBegin_Private(): No of messages: 0
> > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [85] MatStashScatterBegin_Private(): No of messages: 0
> > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [43] MatStashScatterBegin_Private(): No of messages: 0
> > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [5] MatStashScatterBegin_Private(): No of messages: 0
> > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [184] MatStashScatterBegin_Private(): No of messages: 0
> > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [106] MatStashScatterBegin_Private(): No of messages: 0
> > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [74] MatStashScatterBegin_Private(): No of messages: 0
> > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [151] MatStashScatterBegin_Private(): No of messages: 0
> > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [125] MatStashScatterBegin_Private(): No of messages: 0
> > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [173] MatStashScatterBegin_Private(): No of messages: 0
> > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [39] MatStashScatterBegin_Private(): No of messages: 0
> > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [87] MatStashScatterBegin_Private(): No of messages: 0
> > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [46] MatStashScatterBegin_Private(): No of messages: 0
> > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [8] MatStashScatterBegin_Private(): No of messages: 0
> > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [186] MatStashScatterBegin_Private(): No of messages: 0
> > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [107] MatStashScatterBegin_Private(): No of messages: 0
> > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [75] MatStashScatterBegin_Private(): No of messages: 0
> > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [152] MatStashScatterBegin_Private(): No of messages: 0
> > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [126] MatStashScatterBegin_Private(): No of messages: 0
> > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [175] MatStashScatterBegin_Private(): No of messages: 0
> > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [22] MatStashScatterBegin_Private(): No of messages: 0
> > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [88] MatStashScatterBegin_Private(): No of messages: 0
> > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [49] MatStashScatterBegin_Private(): No of messages: 0
> > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [16] MatStashScatterBegin_Private(): No of messages: 0
> > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [189] MatStashScatterBegin_Private(): No of messages: 0
> > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [108] MatStashScatterBegin_Private(): No of messages: 0
> > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [76] MatStashScatterBegin_Private(): No of messages: 0
> > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [153] MatStashScatterBegin_Private(): No of messages: 0
> > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [127] MatStashScatterBegin_Private(): No of messages: 0
> > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [176] MatStashScatterBegin_Private(): No of messages: 0
> > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [23] MatStashScatterBegin_Private(): No of messages: 0
> > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [89] MatStashScatterBegin_Private(): No of messages: 0
> > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [50] MatStashScatterBegin_Private(): No of messages: 0
> > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [18] MatStashScatterBegin_Private(): No of messages: 0
> > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [191] MatStashScatterBegin_Private(): No of messages: 0
> > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [110] MatStashScatterBegin_Private(): No of messages: 0
> > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [77] MatStashScatterBegin_Private(): No of messages: 0
> > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [154] MatStashScatterBegin_Private(): No of messages: 0
> > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [128] MatStashScatterBegin_Private(): No of messages: 0
> > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [177] MatStashScatterBegin_Private(): No of messages: 0
> > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [27] MatStashScatterBegin_Private(): No of messages: 0
> > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [90] MatStashScatterBegin_Private(): No of messages: 0
> > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [51] MatStashScatterBegin_Private(): No of messages: 0
> > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [19] MatStashScatterBegin_Private(): No of messages: 0
> > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [193] MatStashScatterBegin_Private(): No of messages: 0
> > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [117] MatStashScatterBegin_Private(): No of messages: 0
> > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [78] MatStashScatterBegin_Private(): No of messages: 0
> > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [155] MatStashScatterBegin_Private(): No of messages: 0
> > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [129] MatStashScatterBegin_Private(): No of messages: 0
> > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [178] MatStashScatterBegin_Private(): No of messages: 0
> > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [30] MatStashScatterBegin_Private(): No of messages: 0
> > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [96] MatStashScatterBegin_Private(): No of messages: 0
> > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [52] MatStashScatterBegin_Private(): No of messages: 0
> > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [197] MatStashScatterBegin_Private(): No of messages: 0
> > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [118] MatStashScatterBegin_Private(): No of messages: 0
> > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [79] MatStashScatterBegin_Private(): No of messages: 0
> > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [159] MatStashScatterBegin_Private(): No of messages: 0
> > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [131] MatStashScatterBegin_Private(): No of messages: 0
> > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [179] MatStashScatterBegin_Private(): No of messages: 0
> > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [31] MatStashScatterBegin_Private(): No of messages: 0
> > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [97] MatStashScatterBegin_Private(): No of messages: 0
> > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [57] MatStashScatterBegin_Private(): No of messages: 0
> > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [98] MatStashScatterBegin_Private(): No of messages: 0
> > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [58] MatStashScatterBegin_Private(): No of messages: 0
> > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3238253 unneeded,493497 used
> > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197999 unneeded,533751 used
> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243809 unneeded,487941 used
> > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3243446 unneeded,488258 used
> > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3237359 unneeded,494391 used
> > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3245082 unneeded,486622 used
> > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3217795 unneeded,513955 used
> > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3208633 unneeded,523117 used
> > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3209020 unneeded,522684 used
> > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3216305 unneeded,515445 used
> > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3201643 unneeded,530107 used
> > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3234040 unneeded,497664 used
> > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243435 unneeded,488315 used
> > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3249858 unneeded,481846 used
> > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3207727 unneeded,524023 used
> > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3220647 unneeded,511103 used
> > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197939 unneeded,533811 used
> > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3179441 unneeded,552309 used
> > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3197890 unneeded,533814 used
> > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3200381 unneeded,531369 used
> >
> > <ex1.c><inp.log>
> 
> 


From vijay.gopal.c at gmail.com  Sat May  2 15:49:51 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Sat, 2 May 2015 22:49:51 +0200
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
Message-ID: <CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>

Ok so here goes.

I've attached a tarball with a directory which contains the required files
to compile *ex1.c *
The UNIT_L1_ subroutine generates the nonzero elements of my hamiltonian
given a row, it is contained in the
irpf90.a library. The actual fortran code is quite ugly.
Please tell me if you are able to compile and run the codes.

Thanks a lot,
 Vijay

On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> >
> > Please let me clarify,
> >
> > @Matt:
> > I've followed the proper manner to rapid assembly.
> > That is :
> >   1. preallocation
> >   2. nonzero ownership
> > Despite of having done the proper preallocation (as you can see from the
> output file),
> > the matrix takes ages to assemble, whereas the same code works for a
> larger matrix with (100,000,000) elements taking 2 min to assemble !
> >
> > @Barry:
> > I am not touching any file in the code that I attachd.
> > What i'm doing is that each processor will (according to it's ownership
> range) get it's nonzero entries
> > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their
> respective rows.
> > Again to be clear I"m NOT doing file I/O anywhere.
>
>   Oh sorry, where is the fortran code? I assumed the fortran code was
> reading from a file.
>
>    Based on what you sent yes it should take little time; if you send
> something I could run I could try to see why it would be slow.
>
>   Barry
>
> >
> > Could you please have a look at the files attached.
> >
> >
> >
> >
> >
> > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   You have two hundred processes banging on the same file on the same
> filesystem fighting with each other to read in the matrix. That's like
> hiring two hundred guys to dig a ditch but making them share one shovel;
> you won't get speed up, you'll get speed down.
> >
> >
> >   Make a simple sequential PETSc program that reads in the matrix and
> then store it with MatView() in binary format. Then in your parallel
> program use MatLoad() to read the matrix in (it will take a few seconds to
> read in the matrix in that case).
> >
> >   Barry
> >
> >
> > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > I've successfully diagonalized 100 million dimensional matrix in
> 2hours.
> > >
> > > But, the problem is when i try to diagonailze (similar) a smaller
> metrix of
> > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
> running 20 processors each (similar to the 100 million case), the matrix
> assembly itself
> > > takes 2h !
> > >
> > > Could someone point out the mistakes that i'm making ?
> > >
> > > i attach the source code and the output with the mail.
> > >
> > > thanks,
> > >  Vijay
> > >
> > >
> > > #include <slepceps.h>
> > > #include <petsctime.h>
> > >
> > > #undef __FUNCT__
> > > #define __FUNCT__ "main"
> > > int main(int argc,char **argv)
> > > {
> > >   Mat            A;           /* problem matrix */
> > >   EPS            eps;         /* eigenproblem solver context */
> > >   EPSType        type;
> > >   PetscReal      error,tol,re,im;
> > >   PetscScalar    kr,ki,value[52];
> > >   Vec            xr,xi;
> > >   PetscInt
>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > >   PetscInt       d_nz,o_nz;
> > >   PetscLogDouble t1,t2,tt1,tt2;
> > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > >   PetscErrorCode ierr;
> > >   PetscScalar    eigr;
> > >   PetscScalar    eigi;
> > >   PetscScalar    * data;
> > >   Vec          Vr,Vi;
> > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > >   PetscViewer    viewer;
> > >   PetscBool      ishermitian;
> > >   int            mpiid;
> > >   long int            kk,iii;
> > >   long int            tcountcol,tcol[52];
> > >   float          val[52];
> > >   long int            ntrou=1;
> > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2,
> 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
> 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24,
> 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21,
> 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >   double           xtt[52]   ={-1.0,0.,0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >
> > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
> n=%D\n\n",n);CHKERRQ(ierr);
> > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > >
> > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > >   for (i=Istart; i<Iend; i++) {
> > >       tcountcol=0;
> > >       iii=i+1;
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >       }
> > >       unit_l1_(
> > >             l1,
> > >             l2,
> > >             ktyp,
> > >             &iii,
> > >             xjjxy,
> > >             xjjz ,
> > >             xtt  ,
> > >             &tcountcol,
> > >             &ntrou,
> > >             tcol,
> > >             val);
> > >       for(kk=0;kk<52;kk++){
> > >           value[kk] = val[kk];
> > >           col[kk] = tcol[kk]-1;
> > >       }
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime:
> %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > >       }
> > >       countcol=tcountcol;
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >       }
> > >     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > >       }
> > >   }
> > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix:
> %f\n",tt2-tt1);CHKERRQ(ierr);
> > >
> > >
> > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > >
> > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > >
> > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > >   tol = 1.e-8;
> > >   maxit = 10000000;
> > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > >
> > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
> %f\n",t2-t1);CHKERRQ(ierr);
> > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the
> method: %D\n",its);CHKERRQ(ierr);
> > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
> %s\n\n",type);CHKERRQ(ierr);
> > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
> eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g,
> maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > >
> > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > >   /*
> > >   EPSGetConverged(eps,&nconv);
> > >   if (nconv>0) {
> > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > >     EPSIsHermitian(eps,&ishermitian);
> > >     for (i=0;i<nconv;i++) {
> > >       EPSGetEigenvector(eps,i,xr,xi);
> > >       VecView(xr,viewer);
> > > #if !defined(PETSC_USE_COMPLEX)
> > >       if (!ishermitian) { VecView(xi,viewer); }
> > > #endif
> > >     }
> > >     PetscViewerDestroy(&viewer);
> > >   }
> > >   */
> > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > >   ierr = SlepcFinalize();
> > >   return 0;
> > > }
> > >
> > > OUTPUT:
> > >  Time used to build the matrix: 1914.729022
> > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3238253 unneeded,493497 used
> > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197999 unneeded,533751 used
> > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243809 unneeded,487941 used
> > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3243446 unneeded,488258 used
> > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3237359 unneeded,494391 used
> > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3245082 unneeded,486622 used
> > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3217795 unneeded,513955 used
> > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3208633 unneeded,523117 used
> > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3209020 unneeded,522684 used
> > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3216305 unneeded,515445 used
> > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3201643 unneeded,530107 used
> > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3234040 unneeded,497664 used
> > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243435 unneeded,488315 used
> > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3249858 unneeded,481846 used
> > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3207727 unneeded,524023 used
> > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3220647 unneeded,511103 used
> > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197939 unneeded,533811 used
> > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3179441 unneeded,552309 used
> > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3197890 unneeded,533814 used
> > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3200381 unneeded,531369 used
> > >
> > > <ex1.c><inp.log>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/c109c896/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc_problem.tar.gz
Type: application/x-gzip
Size: 83579 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150502/c109c896/attachment-0001.bin>

From bsmith at mcs.anl.gov  Sat May  2 17:24:18 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 2 May 2015 17:24:18 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
Message-ID: <4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>


  Our stupid mail system blocked the attachment because it contained  irpf90.a   Is this a library generated with https://github.com/scemama/irpf90 ?
You need to email everything that goes into making that library because it is unlikely your .a library will work on my machine. What is this thing and can you just tar up the directory that builds it and I can generate irpf90.a

  Barry

> On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> Ok so here goes.
> 
> I've attached a tarball with a directory which contains the required files to compile ex1.c 
> The UNIT_L1_ subroutine generates the nonzero elements of my hamiltonian given a row, it is contained in the
> irpf90.a library. The actual fortran code is quite ugly.
> Please tell me if you are able to compile and run the codes.
> 
> Thanks a lot,
>  Vijay
> 
> On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> >
> > Please let me clarify,
> >
> > @Matt:
> > I've followed the proper manner to rapid assembly.
> > That is :
> >   1. preallocation
> >   2. nonzero ownership
> > Despite of having done the proper preallocation (as you can see from the output file),
> > the matrix takes ages to assemble, whereas the same code works for a larger matrix with (100,000,000) elements taking 2 min to assemble !
> >
> > @Barry:
> > I am not touching any file in the code that I attachd.
> > What i'm doing is that each processor will (according to it's ownership range) get it's nonzero entries
> > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their respective rows.
> > Again to be clear I"m NOT doing file I/O anywhere.
> 
>   Oh sorry, where is the fortran code? I assumed the fortran code was reading from a file.
> 
>    Based on what you sent yes it should take little time; if you send something I could run I could try to see why it would be slow.
> 
>   Barry
> 
> >
> > Could you please have a look at the files attached.
> >
> >
> >
> >
> >
> > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   You have two hundred processes banging on the same file on the same filesystem fighting with each other to read in the matrix. That's like hiring two hundred guys to dig a ditch but making them share one shovel; you won't get speed up, you'll get speed down.
> >
> >
> >   Make a simple sequential PETSc program that reads in the matrix and then store it with MatView() in binary format. Then in your parallel program use MatLoad() to read the matrix in (it will take a few seconds to read in the matrix in that case).
> >
> >   Barry
> >
> >
> > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > I've successfully diagonalized 100 million dimensional matrix in 2hours.
> > >
> > > But, the problem is when i try to diagonailze (similar) a smaller metrix of
> > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20 processors each (similar to the 100 million case), the matrix assembly itself
> > > takes 2h !
> > >
> > > Could someone point out the mistakes that i'm making ?
> > >
> > > i attach the source code and the output with the mail.
> > >
> > > thanks,
> > >  Vijay
> > >
> > >
> > > #include <slepceps.h>
> > > #include <petsctime.h>
> > >
> > > #undef __FUNCT__
> > > #define __FUNCT__ "main"
> > > int main(int argc,char **argv)
> > > {
> > >   Mat            A;           /* problem matrix */
> > >   EPS            eps;         /* eigenproblem solver context */
> > >   EPSType        type;
> > >   PetscReal      error,tol,re,im;
> > >   PetscScalar    kr,ki,value[52];
> > >   Vec            xr,xi;
> > >   PetscInt       n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > >   PetscInt       d_nz,o_nz;
> > >   PetscLogDouble t1,t2,tt1,tt2;
> > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > >   PetscErrorCode ierr;
> > >   PetscScalar    eigr;
> > >   PetscScalar    eigi;
> > >   PetscScalar    * data;
> > >   Vec          Vr,Vi;
> > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > >   PetscViewer    viewer;
> > >   PetscBool      ishermitian;
> > >   int            mpiid;
> > >   long int            kk,iii;
> > >   long int            tcountcol,tcol[52];
> > >   float          val[52];
> > >   long int            ntrou=1;
> > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >   double           xtt[52]   ={-1.0,0.,0.,
> > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > >
> > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem, n=%D\n\n",n);CHKERRQ(ierr);
> > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > >
> > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > >   for (i=Istart; i<Iend; i++) {
> > >       tcountcol=0;
> > >       iii=i+1;
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >       }
> > >       unit_l1_(
> > >             l1,
> > >             l2,
> > >             ktyp,
> > >             &iii,
> > >             xjjxy,
> > >             xjjz ,
> > >             xtt  ,
> > >             &tcountcol,
> > >             &ntrou,
> > >             tcol,
> > >             val);
> > >       for(kk=0;kk<52;kk++){
> > >           value[kk] = val[kk];
> > >           col[kk] = tcol[kk]-1;
> > >       }
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > >       }
> > >       countcol=tcountcol;
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >       }
> > >     ierr = MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > >       if(i%5 == 0 && mpiid==0){
> > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > >       }
> > >   }
> > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > >
> > >
> > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > >
> > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > >
> > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > >   tol = 1.e-8;
> > >   maxit = 10000000;
> > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > >
> > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used: %f\n",t2-t1);CHKERRQ(ierr);
> > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method: %D\n",its);CHKERRQ(ierr);
> > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method: %s\n\n",type);CHKERRQ(ierr);
> > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > >
> > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > >   /*
> > >   EPSGetConverged(eps,&nconv);
> > >   if (nconv>0) {
> > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > >     EPSIsHermitian(eps,&ishermitian);
> > >     for (i=0;i<nconv;i++) {
> > >       EPSGetEigenvector(eps,i,xr,xi);
> > >       VecView(xr,viewer);
> > > #if !defined(PETSC_USE_COMPLEX)
> > >       if (!ishermitian) { VecView(xi,viewer); }
> > > #endif
> > >     }
> > >     PetscViewerDestroy(&viewer);
> > >   }
> > >   */
> > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > >   ierr = SlepcFinalize();
> > >   return 0;
> > > }
> > >
> > > OUTPUT:
> > >  Time used to build the matrix: 1914.729022
> > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3238253 unneeded,493497 used
> > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197999 unneeded,533751 used
> > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243809 unneeded,487941 used
> > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3243446 unneeded,488258 used
> > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3237359 unneeded,494391 used
> > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3245082 unneeded,486622 used
> > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3217795 unneeded,513955 used
> > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3208633 unneeded,523117 used
> > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3209020 unneeded,522684 used
> > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3216305 unneeded,515445 used
> > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3201643 unneeded,530107 used
> > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3234040 unneeded,497664 used
> > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243435 unneeded,488315 used
> > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3249858 unneeded,481846 used
> > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3207727 unneeded,524023 used
> > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3220647 unneeded,511103 used
> > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197939 unneeded,533811 used
> > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3179441 unneeded,552309 used
> > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3197890 unneeded,533814 used
> > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3200381 unneeded,531369 used
> > >
> > > <ex1.c><inp.log>
> >
> >
> 
> 
> <1_Warning.txt>


From vijay.gopal.c at gmail.com  Sat May  2 17:58:37 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Sun, 3 May 2015 00:58:37 +0200
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
Message-ID: <CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>

You are right ! I am using the IRPF90 tool to write my program in FORTRAN.

To use IRPF90 you can clone the github repository and get the irpf90 binary
from the bin directory.
The code to generate the library that contains UNIT_L1_ can be found at:
https://github.com/v1j4y/slepc_version.git
just clone this repository and do:

*make*
*make irpf90.a*

That's it !

I'm resending you the PETSc files along with the makefile. you'll need the
irpf90.a library to make the *ex1 *executable.

Hopefully this time it'll work.
Please let me know if something goes wrong.

On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Our stupid mail system blocked the attachment because it contained
> irpf90.a   Is this a library generated with
> https://github.com/scemama/irpf90 ?
> You need to email everything that goes into making that library because it
> is unlikely your .a library will work on my machine. What is this thing and
> can you just tar up the directory that builds it and I can generate irpf90.a
>
>   Barry
>
> > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> >
> > Ok so here goes.
> >
> > I've attached a tarball with a directory which contains the required
> files to compile ex1.c
> > The UNIT_L1_ subroutine generates the nonzero elements of my hamiltonian
> given a row, it is contained in the
> > irpf90.a library. The actual fortran code is quite ugly.
> > Please tell me if you are able to compile and run the codes.
> >
> > Thanks a lot,
> >  Vijay
> >
> > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > >
> > > Please let me clarify,
> > >
> > > @Matt:
> > > I've followed the proper manner to rapid assembly.
> > > That is :
> > >   1. preallocation
> > >   2. nonzero ownership
> > > Despite of having done the proper preallocation (as you can see from
> the output file),
> > > the matrix takes ages to assemble, whereas the same code works for a
> larger matrix with (100,000,000) elements taking 2 min to assemble !
> > >
> > > @Barry:
> > > I am not touching any file in the code that I attachd.
> > > What i'm doing is that each processor will (according to it's
> ownership range) get it's nonzero entries
> > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for
> their respective rows.
> > > Again to be clear I"m NOT doing file I/O anywhere.
> >
> >   Oh sorry, where is the fortran code? I assumed the fortran code was
> reading from a file.
> >
> >    Based on what you sent yes it should take little time; if you send
> something I could run I could try to see why it would be slow.
> >
> >   Barry
> >
> > >
> > > Could you please have a look at the files attached.
> > >
> > >
> > >
> > >
> > >
> > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >   You have two hundred processes banging on the same file on the same
> filesystem fighting with each other to read in the matrix. That's like
> hiring two hundred guys to dig a ditch but making them share one shovel;
> you won't get speed up, you'll get speed down.
> > >
> > >
> > >   Make a simple sequential PETSc program that reads in the matrix and
> then store it with MatView() in binary format. Then in your parallel
> program use MatLoad() to read the matrix in (it will take a few seconds to
> read in the matrix in that case).
> > >
> > >   Barry
> > >
> > >
> > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > > I've successfully diagonalized 100 million dimensional matrix in
> 2hours.
> > > >
> > > > But, the problem is when i try to diagonailze (similar) a smaller
> metrix of
> > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
> running 20 processors each (similar to the 100 million case), the matrix
> assembly itself
> > > > takes 2h !
> > > >
> > > > Could someone point out the mistakes that i'm making ?
> > > >
> > > > i attach the source code and the output with the mail.
> > > >
> > > > thanks,
> > > >  Vijay
> > > >
> > > >
> > > > #include <slepceps.h>
> > > > #include <petsctime.h>
> > > >
> > > > #undef __FUNCT__
> > > > #define __FUNCT__ "main"
> > > > int main(int argc,char **argv)
> > > > {
> > > >   Mat            A;           /* problem matrix */
> > > >   EPS            eps;         /* eigenproblem solver context */
> > > >   EPSType        type;
> > > >   PetscReal      error,tol,re,im;
> > > >   PetscScalar    kr,ki,value[52];
> > > >   Vec            xr,xi;
> > > >   PetscInt
>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > > >   PetscInt       d_nz,o_nz;
> > > >   PetscLogDouble t1,t2,tt1,tt2;
> > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > > >   PetscErrorCode ierr;
> > > >   PetscScalar    eigr;
> > > >   PetscScalar    eigi;
> > > >   PetscScalar    * data;
> > > >   Vec          Vr,Vi;
> > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > > >   PetscViewer    viewer;
> > > >   PetscBool      ishermitian;
> > > >   int            mpiid;
> > > >   long int            kk,iii;
> > > >   long int            tcountcol,tcol[52];
> > > >   float          val[52];
> > > >   long int            ntrou=1;
> > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1,
> 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
> 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
> 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20,
> 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >   double           xtt[52]   ={-1.0,0.,0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >
> > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
> n=%D\n\n",n);CHKERRQ(ierr);
> > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > > >
> > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > >   for (i=Istart; i<Iend; i++) {
> > > >       tcountcol=0;
> > > >       iii=i+1;
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >       }
> > > >       unit_l1_(
> > > >             l1,
> > > >             l2,
> > > >             ktyp,
> > > >             &iii,
> > > >             xjjxy,
> > > >             xjjz ,
> > > >             xtt  ,
> > > >             &tcountcol,
> > > >             &ntrou,
> > > >             tcol,
> > > >             val);
> > > >       for(kk=0;kk<52;kk++){
> > > >           value[kk] = val[kk];
> > > >           col[kk] = tcol[kk]-1;
> > > >       }
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid:
> %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > > >       }
> > > >       countcol=tcountcol;
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >       }
> > > >     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > > >       }
> > > >   }
> > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > >
> > > >
> > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > > >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > > >   tol = 1.e-8;
> > > >   maxit = 10000000;
> > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > > >
> > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
> %f\n",t2-t1);CHKERRQ(ierr);
> > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the
> method: %D\n",its);CHKERRQ(ierr);
> > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
> %s\n\n",type);CHKERRQ(ierr);
> > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
> eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition:
> tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > > >   /*
> > > >   EPSGetConverged(eps,&nconv);
> > > >   if (nconv>0) {
> > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > > >     EPSIsHermitian(eps,&ishermitian);
> > > >     for (i=0;i<nconv;i++) {
> > > >       EPSGetEigenvector(eps,i,xr,xi);
> > > >       VecView(xr,viewer);
> > > > #if !defined(PETSC_USE_COMPLEX)
> > > >       if (!ishermitian) { VecView(xi,viewer); }
> > > > #endif
> > > >     }
> > > >     PetscViewerDestroy(&viewer);
> > > >   }
> > > >   */
> > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > > >   ierr = SlepcFinalize();
> > > >   return 0;
> > > > }
> > > >
> > > > OUTPUT:
> > > >  Time used to build the matrix: 1914.729022
> > > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3238253 unneeded,493497 used
> > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197999 unneeded,533751 used
> > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243809 unneeded,487941 used
> > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3243446 unneeded,488258 used
> > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3237359 unneeded,494391 used
> > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3245082 unneeded,486622 used
> > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3217795 unneeded,513955 used
> > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3208633 unneeded,523117 used
> > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3209020 unneeded,522684 used
> > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3216305 unneeded,515445 used
> > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3201643 unneeded,530107 used
> > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3234040 unneeded,497664 used
> > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243435 unneeded,488315 used
> > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3249858 unneeded,481846 used
> > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3207727 unneeded,524023 used
> > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3220647 unneeded,511103 used
> > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197939 unneeded,533811 used
> > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3179441 unneeded,552309 used
> > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues()
> is 0
> > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3197890 unneeded,533814 used
> > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3200381 unneeded,531369 used
> > > >
> > > > <ex1.c><inp.log>
> > >
> > >
> >
> >
> > <1_Warning.txt>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150503/f133b272/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex1.c
Type: text/x-csrc
Size: 6249 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150503/f133b272/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 171 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150503/f133b272/attachment-0001.obj>

From tsariysk at craft-tech.com  Sun May  3 10:14:03 2015
From: tsariysk at craft-tech.com (Ted Sariyski)
Date: Sun, 03 May 2015 11:14:03 -0400
Subject: [petsc-users] =?windows-1252?q?error=3A_cannot_convert_=91bool=92?=
 =?windows-1252?q?_to_=91PetscBool=92_for_argument_=912=92?=
In-Reply-To: <9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
Message-ID: <55463B3B.1030006@craft-tech.com>

Hi,
I am looking at petsc-3.5.3/src/snes/examples/tutorials. All examples, 
except ex12.c, works fine.
ex12.c compiles with error: "... cannot convert ?bool? to ?PetscBool? 
..." (see attachment). Do I miss something, or it's a bug?
Thanks,
--
Ted


/> make ex12
*********************W-a-r-n-i-n-g*************************
Your PETSC_DIR may not match the directory you are in
PETSC_DIR  /home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg 
Current directory 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials
Ignore this if you are running make test
******************************************************
mpicxx -o ex12.o -c -Wall -Wwrite-strings -Wno-strict-aliasing 
-Wno-unknown-pragmas -g -O0   -fPIC 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/metis/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include 
-I/opt/mpich-3.1.4/include    `pwd`/ex12.c
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c: 
In function ?PetscErrorCode SetupDiscretization(DM, AppCtx*)?:
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c:505:155: 
error: cannot convert ?bool? to ?PetscBool? for argument ?2? to 
?PetscErrorCode DMPlexAddBoundary(DM, PetscBool, const char*, const 
char*, PetscInt, void (*)(), PetscInt, const PetscInt*, void*)?
make: *** [ex12.o] Error 1


ENVIRONMENT:
---------------------
Linux  2.6.18-128.1.1.el5 #1 SMP Tue Feb 10 11:36:29 EST 2009 x86_64 
x86_64 x86_64 GNU/Linux
g++ (GCC) 4.7.1
mpich-3.1.4

BUILD:
---------
export PETSC_ARCH=petsc-complex-dbg
export PETSC_CMAKE_ARCH=${PETSC_ARCH}
./configure                                        \
PETSC_DIR=${PETSC_DIR}  PETSC_ARCH=${PETSC_ARCH}   \
--prefix=${PETSC_INSTALL_DIR}/${PETSC_ARCH}        \
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 \
--CFLAGS=-O                             \
--with-x                                \
--with-precision=double                 \
--with-scalar-type=complex               \
--with-clanguage=c++                    \
--with-debugger=gdb                       \
--with-debugging                            \
--with-fortran-datatypes=1                 \
--with-metis-dir=${PKGSDIR}/metis         \
--download-f2cblaslapack=yes \
--download-hdf5=yes \
--with-valgrind=1


From bsmith at mcs.anl.gov  Sun May  3 11:04:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 3 May 2015 11:04:57 -0500
Subject: [petsc-users]
 =?windows-1252?q?error=3A_cannot_convert_=91bool=92?=
 =?windows-1252?q?_to_=91PetscBool=92_for_argument_=912=92?=
In-Reply-To: <55463B3B.1030006@craft-tech.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<55463B3B.1030006@craft-tech.com>
Message-ID: <BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>


> On May 3, 2015, at 10:14 AM, Ted Sariyski <tsariysk at craft-tech.com> wrote:
> 
> Hi,
> I am looking at petsc-3.5.3/src/snes/examples/tutorials. All examples, except ex12.c, works fine.
> ex12.c compiles with error: "... cannot convert ?bool? to ?PetscBool? ..." (see attachment). Do I miss something, or it's a bug?

  It is a bug. Currently PetscBool is an integer so a bool (for example (a == 2)) cannot be passed in when a PetscBool is expected. One needs to use 
((a == 2) ? PETSC_TRUE: PETSC_FALSE) to convert the bool to a PetscBool.

  Barry

  Note that PETSc supports C89 which does not have a bool and compiling as C++ code which handles bools slightly differently than C which is why we haven't just converted over to using a bool and dropped the PetscBool. Hopefully eventually we'll be able to drop the PetscBool.


> Thanks,
> --
> Ted
> 
> 
> /> make ex12
> *********************W-a-r-n-i-n-g*************************
> Your PETSC_DIR may not match the directory you are in
> PETSC_DIR  /home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg Current directory /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials
> Ignore this if you are running make test
> ******************************************************
> mpicxx -o ex12.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0   -fPIC -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/home/tsariysk/MFEMv3/pkgs/metis/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/opt/mpich-3.1.4/include    `pwd`/ex12.c
> /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c: In function ?PetscErrorCode SetupDiscretization(DM, AppCtx*)?:
> /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c:505:155: error: cannot convert ?bool? to ?PetscBool? for argument ?2? to ?PetscErrorCode DMPlexAddBoundary(DM, PetscBool, const char*, const char*, PetscInt, void (*)(), PetscInt, const PetscInt*, void*)?
> make: *** [ex12.o] Error 1
> 
> 
> ENVIRONMENT:
> ---------------------
> Linux  2.6.18-128.1.1.el5 #1 SMP Tue Feb 10 11:36:29 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
> g++ (GCC) 4.7.1
> mpich-3.1.4
> 
> BUILD:
> ---------
> export PETSC_ARCH=petsc-complex-dbg
> export PETSC_CMAKE_ARCH=${PETSC_ARCH}
> ./configure                                        \
> PETSC_DIR=${PETSC_DIR}  PETSC_ARCH=${PETSC_ARCH}   \
> --prefix=${PETSC_INSTALL_DIR}/${PETSC_ARCH}        \
> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 \
> --CFLAGS=-O                             \
> --with-x                                \
> --with-precision=double                 \
> --with-scalar-type=complex               \
> --with-clanguage=c++                    \
> --with-debugger=gdb                       \
> --with-debugging                            \
> --with-fortran-datatypes=1                 \
> --with-metis-dir=${PKGSDIR}/metis         \
> --download-f2cblaslapack=yes \
> --download-hdf5=yes \
> --with-valgrind=1
> 


From tsariysk at craft-tech.com  Sun May  3 13:05:32 2015
From: tsariysk at craft-tech.com (Ted Sariyski)
Date: Sun, 03 May 2015 14:05:32 -0400
Subject: [petsc-users]
 =?windows-1252?q?error=3A_cannot_convert_=91bool=92?=
 =?windows-1252?q?_to_=91PetscBool=92_for_argument_=912=92?=
In-Reply-To: <BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<55463B3B.1030006@craft-tech.com>
	<BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>
Message-ID: <5546636C.9000004@craft-tech.com>

Thanks!
--Ted

On 05/03/2015 12:04 PM, Barry Smith wrote:
>> On May 3, 2015, at 10:14 AM, Ted Sariyski <tsariysk at craft-tech.com> wrote:
>>
>> Hi,
>> I am looking at petsc-3.5.3/src/snes/examples/tutorials. All examples, except ex12.c, works fine.
>> ex12.c compiles with error: "... cannot convert ?bool? to ?PetscBool? ..." (see attachment). Do I miss something, or it's a bug?
>    It is a bug. Currently PetscBool is an integer so a bool (for example (a == 2)) cannot be passed in when a PetscBool is expected. One needs to use
> ((a == 2) ? PETSC_TRUE: PETSC_FALSE) to convert the bool to a PetscBool.
>
>    Barry
>
>    Note that PETSc supports C89 which does not have a bool and compiling as C++ code which handles bools slightly differently than C which is why we haven't just converted over to using a bool and dropped the PetscBool. Hopefully eventually we'll be able to drop the PetscBool.
>
>
>> Thanks,
>> --
>> Ted
>>
>>
>> /> make ex12
>> *********************W-a-r-n-i-n-g*************************
>> Your PETSC_DIR may not match the directory you are in
>> PETSC_DIR  /home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg Current directory /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials
>> Ignore this if you are running make test
>> ******************************************************
>> mpicxx -o ex12.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0   -fPIC -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/home/tsariysk/MFEMv3/pkgs/metis/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg/include -I/opt/mpich-3.1.4/include    `pwd`/ex12.c
>> /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c: In function ?PetscErrorCode SetupDiscretization(DM, AppCtx*)?:
>> /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex12.c:505:155: error: cannot convert ?bool? to ?PetscBool? for argument ?2? to ?PetscErrorCode DMPlexAddBoundary(DM, PetscBool, const char*, const char*, PetscInt, void (*)(), PetscInt, const PetscInt*, void*)?
>> make: *** [ex12.o] Error 1
>>
>>
>> ENVIRONMENT:
>> ---------------------
>> Linux  2.6.18-128.1.1.el5 #1 SMP Tue Feb 10 11:36:29 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
>> g++ (GCC) 4.7.1
>> mpich-3.1.4
>>
>> BUILD:
>> ---------
>> export PETSC_ARCH=petsc-complex-dbg
>> export PETSC_CMAKE_ARCH=${PETSC_ARCH}
>> ./configure                                        \
>> PETSC_DIR=${PETSC_DIR}  PETSC_ARCH=${PETSC_ARCH}   \
>> --prefix=${PETSC_INSTALL_DIR}/${PETSC_ARCH}        \
>> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 \
>> --CFLAGS=-O                             \
>> --with-x                                \
>> --with-precision=double                 \
>> --with-scalar-type=complex               \
>> --with-clanguage=c++                    \
>> --with-debugger=gdb                       \
>> --with-debugging                            \
>> --with-fortran-datatypes=1                 \
>> --with-metis-dir=${PKGSDIR}/metis         \
>> --download-f2cblaslapack=yes \
>> --download-hdf5=yes \
>> --with-valgrind=1
>>


From tsariysk at craft-tech.com  Sun May  3 13:15:47 2015
From: tsariysk at craft-tech.com (Ted Sariyski)
Date: Sun, 03 May 2015 14:15:47 -0400
Subject: [petsc-users] [0]PETSC ERROR: Zero pivot in LU factorization:
In-Reply-To: <BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<55463B3B.1030006@craft-tech.com>
	<BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>
Message-ID: <554665D3.3040702@craft-tech.com>

Hi,
I successfully build a version of PETSc with scalar-type=complex. A 
version with the only change of scalar-type=real, builds, but the test 
fails. What do I miss here?
Thanks,
--
Ted


BUILD:
--------
./configure                                            \
PETSC_DIR=${PETSC_DIST_DIR}  PETSC_ARCH=${PETSC_ARCH}  \
--prefix=${PETSC_INSTALL_DIR}/${PETSC_ARCH}            \
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90     \
--CFLAGS=-O                                                        \
--with-x      \
--with-precision=double                                       \
--with-scalar-type=${SCALAR_TYPE}                   \
--with-clanguage=c++                                         \
--with-debugger=gdb                                            \
--with-debugging                                                   \
--with-fortran-datatypes=1                                   \
--with-cgns-dir=${PKGSDIR}/cgns                        \
--with-blas-lapack-dir=${PETSC_DIST_DIR}/${PETSC_ARCH} \
--with-hdf5-dir=${PETSC_DIST_DIR}/${PETSC_ARCH}             \
--download-metis=yes                           \
--download-parmetis=yes                      \
--download-hypre=yes                           \        <------ this is 
not included in the complex build
--with-valgrind=1



TEST
---------------------------------
-bash-3.2$ make 
PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg 
PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg 
and PETSC_ARCH=
*******************Error detected during compile or 
link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials ex19
********************************************************************************* 

*********************W-a-r-n-i-n-g*************************
Your PETSC_DIR may not match the directory you are in
PETSC_DIR  /home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg Current 
directory 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials
Ignore this if you are running make test
******************************************************
mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing 
-Wno-unknown-pragmas -g -O0   -fPIC 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/cgns/include -I/opt/mpich-3.1.4/include 
`pwd`/ex19.c
mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas 
-g -O0   -o ex19  ex19.o 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib  -lpetsc 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lHYPRE 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -llapack -lblas 
-lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/cgns/lib 
-L/home/tsariysk/MFEMv3/pkgs/cgns/lib -lcgns 
-Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib 
-Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 
-Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -lmpifort -lgfortran 
-lm -lgfortran -lm -lmpicxx -lstdc++ -Wl,-rpath,/opt/mpich-3.1.4/lib 
-L/opt/mpich-3.1.4/lib 
-Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 
-Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -ldl 
-Wl,-rpath,/opt/mpich-3.1.4/lib -lmpi -lgcc_s -ldl
/usr/bin/ld: warning: libgfortran.so.1, needed by 
/usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
/usr/bin/ld: warning: libgfortran.so.1, needed by 
/usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
/bin/rm -f ex19.o
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI 
process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Zero pivot in LU factorization: 
http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
[0]PETSC ERROR: Zero pivot, row 2
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015
[0]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com 
by tsariysk Sun May  3 14:02:08 2015
[0]PETSC ERROR: Configure options 
PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 
PETSC_ARCH=petsc-complex-dbg 
--prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg 
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x 
--with-precision=double --with-scalar-type=complex --with-clanguage=c++ 
--with-debugger=gdb --with-debugging --with-fortran-datatypes=1 
--with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
[0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
[0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
[0]PETSC ERROR: #3 MatSOR() line 3643 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 PCApply_SOR() line 35 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
[0]PETSC ERROR: #5 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #6 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: #7 KSPInitialResidual() line 56 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #8 KSPSolve_GMRES() line 234 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #9 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 368 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: #11 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #13 PCApply_MG() line 337 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #14 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #15 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: #16 KSPFGMRESCycle() line 168 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c 

[0]PETSC ERROR: #17 KSPSolve_FGMRES() line 295 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c 

[0]PETSC ERROR: #18 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 232 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #20 SNESSolve() line 3743 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
[0]PETSC ERROR: #21 main() line 158 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 24633 RUNNING AT odin.craft-tech.com
=   EXIT CODE: 71
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI 
processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Zero pivot in LU factorization: 
http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
[0]PETSC ERROR: Zero pivot, row 2
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[1]PETSC ERROR: Zero pivot in LU factorization: 
http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
[1]PETSC ERROR: Zero pivot, row 2
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015
[1]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com 
by tsariysk Sun May  3 14:02:08 2015
[1]PETSC ERROR: Configure options 
PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 
PETSC_ARCH=petsc-complex-dbg 
--prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg 
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x 
--with-precision=double --with-scalar-type=complex --with-clanguage=c++ 
--with-debugger=gdb --with-debugging --with-fortran-datatypes=1 
--with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
[1]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
[1]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
[1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c
Petsc Release Version 3.5.3, Jan, 31, 2015
[0]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com 
by tsariysk Sun May  3 14:02:08 2015
[0]PETSC ERROR: Configure options 
PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 
PETSC_ARCH=petsc-complex-dbg 
--prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg 
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x 
--with-precision=double --with-scalar-type=complex --with-clanguage=c++ 
--with-debugger=gdb --with-debugging --with-fortran-datatypes=1 
--with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg 
--with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
[0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
[0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
[0]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #4 MatSOR() line 3643 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
[1]PETSC ERROR: [0]PETSC ERROR: #4 MatSOR() line 3643 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
#5 PCApply_SOR() line 35 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
[1]PETSC ERROR: [0]PETSC ERROR: #5 PCApply_SOR() line 35 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
[0]PETSC ERROR: #6 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
#6 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: [0]PETSC ERROR: #7 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
#7 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: #8 KSPInitialResidual() line 56 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
[1]PETSC ERROR: #8 KSPInitialResidual() line 56 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #9 KSPSolve_GMRES() line 234 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: [1]PETSC ERROR: #9 KSPSolve_GMRES() line 234 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
#10 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #11 KSPSolve_Chebyshev() line 368 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: [1]PETSC ERROR: #10 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#12 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [1]PETSC ERROR: #11 KSPSolve_Chebyshev() line 368 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
#13 PCMGMCycle_Private() line 19 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #14 PCApply_MG() line 337 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: [1]PETSC ERROR: #12 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#15 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #16 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
[1]PETSC ERROR: #13 PCMGMCycle_Private() line 19 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #17 KSPFGMRESCycle() line 168 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: #18 KSPSolve_FGMRES() line 295 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[1]PETSC ERROR: #14 PCApply_MG() line 337 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #19 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #20 SNESSolve_NEWTONLS() line 232 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #15 PCApply() line 440 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #21 SNESSolve() line 3743 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
[1]PETSC ERROR: #16 KSP_PCApply() line 230 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: #22 main() line 158 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: #17 KSPFGMRESCycle() line 168 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[1]PETSC ERROR: #18 KSPSolve_FGMRES() line 295 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[1]PETSC ERROR: #19 KSPSolve() line 460 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
[1]PETSC ERROR: #20 SNESSolve_NEWTONLS() line 232 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #21 SNESSolve() line 3743 in 
/work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
[1]PETSC ERROR: #22 main() line 158 in 
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
[1]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 71) - process 1

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 24638 RUNNING AT odin.craft-tech.com
=   EXIT CODE: 71
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
*******************Error detected during compile or link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials ex5f90t
*********************************************************
mpif90 -c  -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 
-Wno-unused-dummy-argument -g -O0 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include 
-I/home/tsariysk/MFEMv3/pkgs/cgns/include -I/opt/mpich-3.1.4/include    
-o ex5f90t.o ex5f90t.F
mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 
-Wno-unused-dummy-argument -g -O0   -o ex5f90t ex5f90t.o 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib  -lpetsc 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lHYPRE 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg 
-L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -llapack -lblas 
-lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib 
-lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 
-Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/cgns/lib 
-L/home/tsariysk/MFEMv3/pkgs/cgns/lib -lcgns 
-Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib 
-Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 
-Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -lmpifort -lgfortran 
-lm -lgfortran -lm -lmpicxx -lstdc++ -Wl,-rpath,/opt/mpich-3.1.4/lib 
-L/opt/mpich-3.1.4/lib 
-Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 
-Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 
-Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -ldl 
-Wl,-rpath,/opt/mpich-3.1.4/lib -lmpi -lgcc_s -ldl
/usr/bin/ld: warning: libgfortran.so.1, needed by 
/usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
/usr/bin/ld: warning: libgfortran.so.1, needed by 
/usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
/usr/bin/ld: warning: libgfortran.so.1, needed by 
/usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
/bin/rm -f ex5f90t.o
Possible error running Fortran example 
src/snes/examples/tutorials/ex5f90t with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
Number of SNES iterations =     0
Completed test examples
=========================================
Now to evaluate the computer systems you plan use - do:
make PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg 
PETSC_ARCH= streams NPMAX=<number of MPI processes you intend to use>


From bsmith at mcs.anl.gov  Sun May  3 13:36:19 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 3 May 2015 13:36:19 -0500
Subject: [petsc-users] [0]PETSC ERROR: Zero pivot in LU factorization:
In-Reply-To: <554665D3.3040702@craft-tech.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<55463B3B.1030006@craft-tech.com>
	<BE5B94DE-0D28-4DE2-A5B9-F1DA044CD746@mcs.anl.gov>
	<554665D3.3040702@craft-tech.com>
Message-ID: <5AA7C57E-832F-4EEB-AB5C-949119B487F0@mcs.anl.gov>


   There is some mixup between your complex and real build and the test examples. When ex19 fails it prints 

> [0]PETSC ERROR: Configure options PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 PETSC_ARCH=petsc-complex-dbg --prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x --with-precision=double --with-scalar-type=complex --with-clanguage=c++ --with-debugger=gdb --with-debugging --with-fortran-datatypes=1 --with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1

  note all the petsc-complex-dbg. Yet it was built allegedly with the real version so should not list these things.  I cannot explain how it got into this state.

   I would start all over again with a minimal set of configure options: no CFLAGS, no --with-clanguage no hypre, metis, parmetis, 

  Barry


> On May 3, 2015, at 1:15 PM, Ted Sariyski <tsariysk at craft-tech.com> wrote:
> 
> Hi,
> I successfully build a version of PETSc with scalar-type=complex. A version with the only change of scalar-type=real, builds, but the test fails. What do I miss here?
> Thanks,
> --
> Ted
> 
> 
> BUILD:
> --------
> ./configure                                            \
> PETSC_DIR=${PETSC_DIST_DIR}  PETSC_ARCH=${PETSC_ARCH}  \
> --prefix=${PETSC_INSTALL_DIR}/${PETSC_ARCH}            \
> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90     \
> --CFLAGS=-O                                                        \
> --with-x      \
> --with-precision=double                                       \
> --with-scalar-type=${SCALAR_TYPE}                   \
> --with-clanguage=c++                                         \
> --with-debugger=gdb                                            \
> --with-debugging                                                   \
> --with-fortran-datatypes=1                                   \
> --with-cgns-dir=${PKGSDIR}/cgns                        \
> --with-blas-lapack-dir=${PETSC_DIST_DIR}/${PETSC_ARCH} \
> --with-hdf5-dir=${PETSC_DIST_DIR}/${PETSC_ARCH}             \
> --download-metis=yes                           \
> --download-parmetis=yes                      \
> --download-hypre=yes                           \        <------ this is not included in the complex build
> --with-valgrind=1
> 
> 
> 
> TEST
> ---------------------------------
> -bash-3.2$ make PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg PETSC_ARCH="" test
> Running test examples to verify correct installation
> Using PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg and PETSC_ARCH=
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials ex19
> ********************************************************************************* 
> *********************W-a-r-n-i-n-g*************************
> Your PETSC_DIR may not match the directory you are in
> PETSC_DIR  /home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg Current directory /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials
> Ignore this if you are running make test
> ******************************************************
> mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0   -fPIC -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/cgns/include -I/opt/mpich-3.1.4/include `pwd`/ex19.c
> mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0   -o ex19  ex19.o -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib  -lpetsc -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lHYPRE -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -llapack -lblas -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/cgns/lib -L/home/tsariysk/MFEMv3/pkgs/cgns/lib -lcgns -Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib -Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 -Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -lmpifort -lgfortran -lm -lgfortran -lm -lmpicxx -lstdc++ -Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib -Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 -Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -ldl -Wl,-rpath,/opt/mpich-3.1.4/lib -lmpi -lgcc_s -ldl
> /usr/bin/ld: warning: libgfortran.so.1, needed by /usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
> /usr/bin/ld: warning: libgfortran.so.1, needed by /usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
> /bin/rm -f ex19.o
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
> [0]PETSC ERROR: Zero pivot, row 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015
> [0]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com by tsariysk Sun May  3 14:02:08 2015
> [0]PETSC ERROR: Configure options PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 PETSC_ARCH=petsc-complex-dbg --prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x --with-precision=double --with-scalar-type=complex --with-clanguage=c++ --with-debugger=gdb --with-debugging --with-fortran-datatypes=1 --with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
> [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
> [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
> [0]PETSC ERROR: #3 MatSOR() line 3643 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 PCApply_SOR() line 35 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
> [0]PETSC ERROR: #5 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #6 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> [0]PETSC ERROR: #7 KSPInitialResidual() line 56 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 234 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #9 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #10 KSPSolve_Chebyshev() line 368 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #11 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #12 PCMGMCycle_Private() line 19 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #13 PCApply_MG() line 337 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #14 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #15 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> [0]PETSC ERROR: #16 KSPFGMRESCycle() line 168 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c 
> [0]PETSC ERROR: #17 KSPSolve_FGMRES() line 295 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c 
> [0]PETSC ERROR: #18 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #19 SNESSolve_NEWTONLS() line 232 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #20 SNESSolve() line 3743 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
> [0]PETSC ERROR: #21 main() line 158 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> 
> ===================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   PID 24633 RUNNING AT odin.craft-tech.com
> =   EXIT CODE: 71
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> ===================================================================================
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
> [0]PETSC ERROR: Zero pivot, row 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
> [1]PETSC ERROR: Zero pivot, row 2
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015
> [1]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com by tsariysk Sun May  3 14:02:08 2015
> [1]PETSC ERROR: Configure options PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 PETSC_ARCH=petsc-complex-dbg --prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x --with-precision=double --with-scalar-type=complex --with-clanguage=c++ --with-debugger=gdb --with-debugging --with-fortran-datatypes=1 --with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
> [1]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
> [1]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
> [1]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c
> Petsc Release Version 3.5.3, Jan, 31, 2015
> [0]PETSC ERROR: ./ex19 on a petsc-complex-dbg named odin.craft-tech.com by tsariysk Sun May  3 14:02:08 2015
> [0]PETSC ERROR: Configure options PETSC_DIR=/home/tsariysk/MFEMv3/tars/petsc-3.5.3 PETSC_ARCH=petsc-complex-dbg --prefix=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-complex-dbg --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --CFLAGS=-O --with-x --with-precision=double --with-scalar-type=complex --with-clanguage=c++ --with-debugger=gdb --with-debugging --with-fortran-datatypes=1 --with-blas-lapack-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-hdf5-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-metis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-parmetis-dir=/home/tsariysk/MFEMv3/tars/petsc-3.5.3/petsc-complex-dbg --with-cgns-dir=/home/tsariysk/MFEMv3/pkgs/cgns --with-valgrind=1
> [0]PETSC ERROR: #1 PetscKernel_A_gets_inverse_A_4() line 48 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/baij/seq/dgefa4.c
> [0]PETSC ERROR: #2 MatSOR_SeqAIJ_Inode() line 2812 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/seq/inode.c
> [0]PETSC ERROR: #3 MatSOR_MPIAIJ() line 1483 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #4 MatSOR() line 3643 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
> [1]PETSC ERROR: [0]PETSC ERROR: #4 MatSOR() line 3643 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/mat/interface/matrix.c
> #5 PCApply_SOR() line 35 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
> [1]PETSC ERROR: [0]PETSC ERROR: #5 PCApply_SOR() line 35 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/sor/sor.c
> [0]PETSC ERROR: #6 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> #6 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [0]PETSC ERROR: #7 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> #7 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> [0]PETSC ERROR: #8 KSPInitialResidual() line 56 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
> [1]PETSC ERROR: #8 KSPInitialResidual() line 56 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #9 KSPSolve_GMRES() line 234 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: [1]PETSC ERROR: #9 KSPSolve_GMRES() line 234 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c
> #10 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #11 KSPSolve_Chebyshev() line 368 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: [1]PETSC ERROR: #10 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> #12 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [1]PETSC ERROR: #11 KSPSolve_Chebyshev() line 368 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/cheby/cheby.c
> #13 PCMGMCycle_Private() line 19 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #14 PCApply_MG() line 337 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: [1]PETSC ERROR: #12 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> #15 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #16 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> [1]PETSC ERROR: #13 PCMGMCycle_Private() line 19 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #17 KSPFGMRESCycle() line 168 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #18 KSPSolve_FGMRES() line 295 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [1]PETSC ERROR: #14 PCApply_MG() line 337 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #19 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #20 SNESSolve_NEWTONLS() line 232 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #15 PCApply() line 440 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #21 SNESSolve() line 3743 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
> [1]PETSC ERROR: #16 KSP_PCApply() line 230 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/include/petsc-private/kspimpl.h
> [0]PETSC ERROR: #22 main() line 158 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: #17 KSPFGMRESCycle() line 168 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [1]PETSC ERROR: #18 KSPSolve_FGMRES() line 295 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [1]PETSC ERROR: #19 KSPSolve() line 460 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
> application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> [1]PETSC ERROR: #20 SNESSolve_NEWTONLS() line 232 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #21 SNESSolve() line 3743 in /work/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/interface/snes.c
> [1]PETSC ERROR: #22 main() line 158 in /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials/ex19.c
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 71) - process 1
> 
> ===================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   PID 24638 RUNNING AT odin.craft-tech.com
> =   EXIT CODE: 71
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> ===================================================================================
> *******************Error detected during compile or link!*******************
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /home/tsariysk/MFEMv3/tars/petsc-3.5.3/src/snes/examples/tutorials ex5f90t
> *********************************************************
> mpif90 -c  -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0 -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/include -I/home/tsariysk/MFEMv3/pkgs/cgns/include -I/opt/mpich-3.1.4/include    -o ex5f90t.o ex5f90t.F
> mpif90 -fPIC -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -g -O0   -o ex5f90t ex5f90t.o -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib  -lpetsc -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lHYPRE -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -L/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg -llapack -lblas -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -Wl,-rpath,/home/tsariysk/MFEMv3/pkgs/cgns/lib -L/home/tsariysk/MFEMv3/pkgs/cgns/lib -lcgns -Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib -Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 -Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -lmpifort -lgfortran -lm -lgfortran -lm -lmpicxx -lstdc++ -Wl,-rpath,/opt/mpich-3.1.4/lib -L/opt/mpich-3.1.4/lib -Wl,-rpath,/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/opt/gcc/4.7.1/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -Wl,-rpath,/opt/gcc/4.7.1/lib64 -L/opt/gcc/4.7.1/lib64 -Wl,-rpath,/opt/gcc/4.7.1/lib -L/opt/gcc/4.7.1/lib -ldl -Wl,-rpath,/opt/mpich-3.1.4/lib -lmpi -lgcc_s -ldl
> /usr/bin/ld: warning: libgfortran.so.1, needed by /usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
> /usr/bin/ld: warning: libgfortran.so.1, needed by /usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
> /usr/bin/ld: warning: libgfortran.so.1, needed by /usr/lib/../lib64/liblapack.so, may conflict with libgfortran.so.3
> /bin/rm -f ex5f90t.o
> Possible error running Fortran example src/snes/examples/tutorials/ex5f90t with 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> Number of SNES iterations =     0
> Completed test examples
> =========================================
> Now to evaluate the computer systems you plan use - do:
> make PETSC_DIR=/home/tsariysk/MFEMv3/pkgs/petsc-3.5.3/petsc-real-dbg PETSC_ARCH= streams NPMAX=<number of MPI processes you intend to use>
> 


From victorsv at gmail.com  Mon May  4 02:42:03 2015
From: victorsv at gmail.com (victor sv)
Date: Mon, 4 May 2015 09:42:03 +0200
Subject: [petsc-users] [Petsc] Command Line Arguments
Message-ID: <CA+Wz_FwB49qB2eEbYG5RCW0RTzRL_B2mDzhggirqrABhrm4ftw@mail.gmail.com>

 Hello,

I need to implement/integrate a command line interface (CLI) into a project.


I have heard a lot of good things about the PETSc CLI and also I have found
this old post talking a little about it:

http://www.cfd-online.com/Forums/main/62946-trilinos-petsc-better.html

I was looking for some information about PETSc CLI architecture or design
pattern but i didn't find any information. I also have been diving for the
code, but  is not simple..

    Is there any document describing the architecture of PETSc CLI? Can I
have access to that information?

Thanks in advance and congratulations for your great job with PETSc.

Best regards,
V?ctor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/5050aea3/attachment.html>

From jychang48 at gmail.com  Mon May  4 07:36:04 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 4 May 2015 07:36:04 -0500
Subject: [petsc-users] Obtaining bytes per second
Message-ID: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>

Hello everyone,

If I wanted to obtain the bytes/second for my PETSc program, is there a
generic way of doing this? My initial thought would be to first run the
program with valgrind to obtain the total memory usage, and then run it
without valgrind to get the wall clock time. These two metrics then give
you the bytes/second.

Or can PETSc manually count the load/stores the way it's done for flops? I
was looking at the PetscMemXXX() functions but wasn't sure if this is what
I was looking for.

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/bd6d56a1/attachment.html>

From jed at jedbrown.org  Mon May  4 08:05:39 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 04 May 2015 07:05:39 -0600
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
Message-ID: <87fv7cjxwc.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Hello everyone,
>
> If I wanted to obtain the bytes/second for my PETSc program, is there a
> generic way of doing this? My initial thought would be to first run the
> program with valgrind to obtain the total memory usage, and then run it
> without valgrind to get the wall clock time. These two metrics then give
> you the bytes/second.

Not really, because usually we're interested in useful bandwidth
sustained from some level of cache.  You can use hardware performance
counters to measure the number of cache lines transferred, but this is
usually an overestimate of the amount of useful data.  You really need a
performance model for your application and a cache model for the machine
to say what bandwidth is useful.

> Or can PETSc manually count the load/stores the way it's done for
> flops? 

No, this information is not available in source code and would be nearly
meaningless even if it was.

> I was looking at the PetscMemXXX() functions but wasn't sure if this
> is what I was looking for.
>
> Thanks,
> Justin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/85ddfa99/attachment.pgp>

From jychang48 at gmail.com  Mon May  4 09:07:55 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 4 May 2015 09:07:55 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87fv7cjxwc.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
Message-ID: <CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>

Hi Jed,

Thanks for the reply. Not too long ago one of you guys (Matt I think) had
mentioned the Roofline model and I was hoping to emulate something like it
for my application. If I understand the presentation slides (and the paper
implementing it) correctly, the upper bound FLOPS/s is calculated by
multiplying the stream BW by the ratio of DRAM flop to byte (aka arithmetic
intensity). The workload (i.e., flops) can be counted via PetscLogFlops()
and in the paper, the sparse matvec total bytes transferred for fmadd was
manually counted. Since my program involves more than just matvec I am
curious if there's a way to obtain the bytes for all operations and
functions invoked.

Or if I really should go with what you had suggested, could you elaborate a
little more on it, or point me to some papers/links/slides that talk about
it?

Thanks,
Justin

On Monday, May 4, 2015, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > Hello everyone,
> >
> > If I wanted to obtain the bytes/second for my PETSc program, is there a
> > generic way of doing this? My initial thought would be to first run the
> > program with valgrind to obtain the total memory usage, and then run it
> > without valgrind to get the wall clock time. These two metrics then give
> > you the bytes/second.
>
> Not really, because usually we're interested in useful bandwidth
> sustained from some level of cache.  You can use hardware performance
> counters to measure the number of cache lines transferred, but this is
> usually an overestimate of the amount of useful data.  You really need a
> performance model for your application and a cache model for the machine
> to say what bandwidth is useful.
>
> > Or can PETSc manually count the load/stores the way it's done for
> > flops?
>
> No, this information is not available in source code and would be nearly
> meaningless even if it was.
>
> > I was looking at the PetscMemXXX() functions but wasn't sure if this
> > is what I was looking for.
> >
> > Thanks,
> > Justin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/3b77e47c/attachment.html>

From knepley at gmail.com  Mon May  4 11:01:22 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 May 2015 09:01:22 -0700
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
Message-ID: <CAMYG4G=EiFyAxm_SrM4kYdJhnEqMrRNSLZupr6TF5yTyetSThg@mail.gmail.com>

On Mon, May 4, 2015 at 7:07 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Hi Jed,
>
> Thanks for the reply. Not too long ago one of you guys (Matt I think) had
> mentioned the Roofline model and I was hoping to emulate something like it
> for my application. If I understand the presentation slides (and the paper
> implementing it) correctly, the upper bound FLOPS/s is calculated by
> multiplying the stream BW by the ratio of DRAM flop to byte (aka arithmetic
> intensity). The workload (i.e., flops) can be counted via PetscLogFlops()
> and in the paper, the sparse matvec total bytes transferred for fmadd was
> manually counted. Since my program involves more than just matvec I am
> curious if there's a way to obtain the bytes for all operations and
> functions invoked.
>
> Or if I really should go with what you had suggested, could you elaborate
> a little more on it, or point me to some papers/links/slides that talk
> about it?
>

The best we can do is estimates here (because of all the caveats that Jed
points out). I suggest just counting
how many bytes come down manually, just as we do for flops.

  Thanks,

    Matt


> Thanks,
> Justin
>
>
> On Monday, May 4, 2015, Jed Brown <jed at jedbrown.org> wrote:
>
>> Justin Chang <jychang48 at gmail.com> writes:
>>
>> > Hello everyone,
>> >
>> > If I wanted to obtain the bytes/second for my PETSc program, is there a
>> > generic way of doing this? My initial thought would be to first run the
>> > program with valgrind to obtain the total memory usage, and then run it
>> > without valgrind to get the wall clock time. These two metrics then give
>> > you the bytes/second.
>>
>> Not really, because usually we're interested in useful bandwidth
>> sustained from some level of cache.  You can use hardware performance
>> counters to measure the number of cache lines transferred, but this is
>> usually an overestimate of the amount of useful data.  You really need a
>> performance model for your application and a cache model for the machine
>> to say what bandwidth is useful.
>>
>> > Or can PETSc manually count the load/stores the way it's done for
>> > flops?
>>
>> No, this information is not available in source code and would be nearly
>> meaningless even if it was.
>>
>> > I was looking at the PetscMemXXX() functions but wasn't sure if this
>> > is what I was looking for.
>> >
>> > Thanks,
>> > Justin
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/16afdf8a/attachment.html>

From jed at jedbrown.org  Mon May  4 11:07:32 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 04 May 2015 10:07:32 -0600
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
Message-ID: <87bni0jph7.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Hi Jed,
>
> Thanks for the reply. Not too long ago one of you guys (Matt I think) had
> mentioned the Roofline model and I was hoping to emulate something like it
> for my application. If I understand the presentation slides (and the paper
> implementing it) correctly, the upper bound FLOPS/s is calculated by
> multiplying the stream BW by the ratio of DRAM flop to byte (aka arithmetic
> intensity). The workload (i.e., flops) can be counted via PetscLogFlops()
> and in the paper, the sparse matvec total bytes transferred for fmadd was
> manually counted. Since my program involves more than just matvec I am
> curious if there's a way to obtain the bytes for all operations and
> functions invoked.

Counting "useful" data motion subject to some cache granularity is not
automatic.  You can look at performance analysis of stencil operations
for an example of what this can look like.  I go through examples in my
class, but I do it interactively with experiments rather than off of
slides.

> Or if I really should go with what you had suggested, could you elaborate a
> little more on it, or point me to some papers/links/slides that talk about
> it?
>
> Thanks,
> Justin
>
> On Monday, May 4, 2015, Jed Brown <jed at jedbrown.org> wrote:
>
>> Justin Chang <jychang48 at gmail.com> writes:
>>
>> > Hello everyone,
>> >
>> > If I wanted to obtain the bytes/second for my PETSc program, is there a
>> > generic way of doing this? My initial thought would be to first run the
>> > program with valgrind to obtain the total memory usage, and then run it
>> > without valgrind to get the wall clock time. These two metrics then give
>> > you the bytes/second.
>>
>> Not really, because usually we're interested in useful bandwidth
>> sustained from some level of cache.  You can use hardware performance
>> counters to measure the number of cache lines transferred, but this is
>> usually an overestimate of the amount of useful data.  You really need a
>> performance model for your application and a cache model for the machine
>> to say what bandwidth is useful.
>>
>> > Or can PETSc manually count the load/stores the way it's done for
>> > flops?
>>
>> No, this information is not available in source code and would be nearly
>> meaningless even if it was.
>>
>> > I was looking at the PetscMemXXX() functions but wasn't sure if this
>> > is what I was looking for.
>> >
>> > Thanks,
>> > Justin
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/69e92757/attachment-0001.pgp>

From knepley at gmail.com  Mon May  4 11:12:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 May 2015 09:12:58 -0700
Subject: [petsc-users] [Petsc] Command Line Arguments
In-Reply-To: <CA+Wz_FwB49qB2eEbYG5RCW0RTzRL_B2mDzhggirqrABhrm4ftw@mail.gmail.com>
References: <CA+Wz_FwB49qB2eEbYG5RCW0RTzRL_B2mDzhggirqrABhrm4ftw@mail.gmail.com>
Message-ID: <CAMYG4GnV06ehXKxR8fhJKvOG1SrzKYCiNL4XejVNhrrf-ruGiw@mail.gmail.com>

On Mon, May 4, 2015 at 12:42 AM, victor sv <victorsv at gmail.com> wrote:

> Hello,
>
> I need to implement/integrate a command line interface (CLI) into a project.
>
>
> I have heard a lot of good things about the PETSc CLI and also I have
> found this old post talking a little about it:
>
> http://www.cfd-online.com/Forums/main/62946-trilinos-petsc-better.html
>
> I was looking for some information about PETSc CLI architecture or design
> pattern but i didn't find any information. I also have been diving for the
> code, but  is not simple..
>
>     Is there any document describing the architecture of PETSc CLI? Can I
> have access to that information?
>

Hi Victor,

  There is a general discussion of PETSc command line options in Chapter 14
of the manual.

  Thanks,

     Matt


> Thanks in advance and congratulations for your great job with PETSc.
>
> Best regards,
> V?ctor.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/3a4fdcc9/attachment.html>

From stefano.pedraglio at libero.it  Mon May  4 14:34:34 2015
From: stefano.pedraglio at libero.it (stefano.pedraglio at libero.it)
Date: Mon, 4 May 2015 21:34:34 +0200 (CEST)
Subject: [petsc-users] Load real matrix in complex PETSc build.
Message-ID: <327109632.1459071430768074405.JavaMail.httpd@webmail-28.iol.local>

Hallo
I have saved a large matrix using PETSc configured with scalar type real and I used binary viewer. I need now to compute the eigensolutions in another program using SLEPc and I need to use a complex shift. I have configured and compiled another PETSc and SLEPc build for complex scalars. I know it is not possible to load a matrix saved as real when using a program configured for complex numbers. Is there a workaround to perform this operation? Maybe saving and loading in some other format the matrix or using a  different library for eigenvalue computation. Thank you for any suggestion.

Hallo
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/fe299bba/attachment.html>

From bsmith at mcs.anl.gov  Mon May  4 14:40:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 4 May 2015 14:40:23 -0500
Subject: [petsc-users] Load real matrix in complex PETSc build.
In-Reply-To: <327109632.1459071430768074405.JavaMail.httpd@webmail-28.iol.local>
References: <327109632.1459071430768074405.JavaMail.httpd@webmail-28.iol.local>
Message-ID: <0D104B1A-DEFD-4020-81C1-F707F021DF21@mcs.anl.gov>


   You should be able to do it using the PetscBinaryIO.py tools we provide in the bin directory. Just read in the matrix as real in python and then write it out as complex using the methods there.

   Barry

> On May 4, 2015, at 2:34 PM, stefano.pedraglio at libero.it wrote:
> 
> Hallo
> I have saved a large matrix using PETSc configured with scalar type real and I used binary viewer. I need now to compute the eigensolutions in another program using SLEPc and I need to use a complex shift. I have configured and compiled another PETSc and SLEPc build for complex scalars. I know it is not possible to load a matrix saved as real when using a program configured for complex numbers. Is there a workaround to perform this operation? Maybe saving and loading in some other format the matrix or using a  different library for eigenvalue computation. Thank you for any suggestion.
> 
> Hallo
> Stefano


From reza.yaghmaie2 at gmail.com  Mon May  4 16:11:09 2015
From: reza.yaghmaie2 at gmail.com (Reza Yaghmaie)
Date: Mon, 4 May 2015 17:11:09 -0400
Subject: [petsc-users] (no subject)
Message-ID: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>

Dear PETSC representatives,

I am solving a nonlinear problem with SNESNGMRES and it converges faster
with less iterations compared to otehr SNES methods. Any idea why that is
the case?
Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for the
linesearch option and nothing changes.
I presume it is using the default SNESQN method. Btw, there are three
options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN" in
teh manual. I tried to associate them with SNES however it seems these
hyphened names don't work there. What am I missing?

Best regards,
Ray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/272ea41b/attachment.html>

From alpkalpalp at gmail.com  Mon May  4 17:18:35 2015
From: alpkalpalp at gmail.com (Alp Kalpalp)
Date: Tue, 5 May 2015 01:18:35 +0300
Subject: [petsc-users] Getting unsupported type message from
	MatCholeskyFactor
Message-ID: <CALi504T_rby7A7qQYf42TYZgnj8=rX_oKeNAYpSYYemJpDKtZA@mail.gmail.com>

I have tested seqaij and mpiaij types but still getting unspoorted type
error from MatCholeskyFactor function. Should I activate an external
packege like SuperLU ?

Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/b26901e3/attachment.html>

From hzhang at mcs.anl.gov  Mon May  4 17:23:33 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Mon, 4 May 2015 17:23:33 -0500
Subject: [petsc-users] Getting unsupported type message from
	MatCholeskyFactor
In-Reply-To: <CALi504T_rby7A7qQYf42TYZgnj8=rX_oKeNAYpSYYemJpDKtZA@mail.gmail.com>
References: <CALi504T_rby7A7qQYf42TYZgnj8=rX_oKeNAYpSYYemJpDKtZA@mail.gmail.com>
Message-ID: <CAGCphBtz95fn=Bz=gDE4Y9D77nbK79LRptm5OCBo4z5bV9vK0g@mail.gmail.com>

Alp :

> I have tested seqaij and mpiaij types but still getting unspoorted type
> error from MatCholeskyFactor function. Should I activate an external
> packege like SuperLU ?
>

Petsc only supports sequential  Cholesky. You must use MUMPS for parallel
Cholesky.
SuperLU is a sequential; SuperLU_DIST supports parallel LU, not Cholesky.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/7f644bfd/attachment.html>

From knepley at gmail.com  Mon May  4 17:30:17 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 May 2015 15:30:17 -0700
Subject: [petsc-users] (no subject)
In-Reply-To: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
References: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
Message-ID: <CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>

On Mon, May 4, 2015 at 2:11 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
wrote:

>
> Dear PETSC representatives,
>
> I am solving a nonlinear problem with SNESNGMRES and it converges faster
> with less iterations compared to otehr SNES methods. Any idea why that is
> the case?
>

It is impossible to tell with this information.


> Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for the
> linesearch option and nothing changes.
>

This can happen, especially if your matrix is far from the identity.


> I presume it is using the default SNESQN method. Btw, there are three
> options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN" in
> teh manual. I tried to associate them with SNES however it seems these
> hyphened names don't work there. What am I missing?
>

-snes_qn_scale_type <lbfgs,broyden,badbroyden>

from
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetType.html

   Thanks,

    Matt

Best regards,
> Ray
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/e659cbfe/attachment.html>

From reza.yaghmaie2 at gmail.com  Mon May  4 17:41:36 2015
From: reza.yaghmaie2 at gmail.com (Reza Yaghmaie)
Date: Mon, 4 May 2015 18:41:36 -0400
Subject: [petsc-users] (no subject)
In-Reply-To: <CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>
References: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
	<CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>
Message-ID: <CADOqKp=yAaK4s6cPVOgD_y_+tZTfXMGyKDXq8ad0gTE3WXFLyg@mail.gmail.com>

Dear Matt,

Actually the initial jacobian was identity. Regular SNES converges in 48
iterations, GMRES in 19, NCG in 67,...
Do you think SNESQN with the basiclineseach was the problem for divergence?
If I use SNESQN by default should not it converge with initial identity
jacobian?

Best regards,
Reza




On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 4, 2015 at 2:11 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com
> <javascript:_e(%7B%7D,'cvml','reza.yaghmaie2 at gmail.com');>> wrote:
>
>>
>> Dear PETSC representatives,
>>
>> I am solving a nonlinear problem with SNESNGMRES and it converges faster
>> with less iterations compared to otehr SNES methods. Any idea why that is
>> the case?
>>
>
> It is impossible to tell with this information.
>
>
>> Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for the
>> linesearch option and nothing changes.
>>
>
> This can happen, especially if your matrix is far from the identity.
>
>
>> I presume it is using the default SNESQN method. Btw, there are three
>> options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN"
>> in teh manual. I tried to associate them with SNES however it seems these
>> hyphened names don't work there. What am I missing?
>>
>
> -snes_qn_scale_type <lbfgs,broyden,badbroyden>
>
> from
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetType.html
>
>    Thanks,
>
>     Matt
>
> Best regards,
>> Ray
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/9586f915/attachment-0001.html>

From knepley at gmail.com  Mon May  4 17:53:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 May 2015 17:53:19 -0500
Subject: [petsc-users] (no subject)
In-Reply-To: <CADOqKp=yAaK4s6cPVOgD_y_+tZTfXMGyKDXq8ad0gTE3WXFLyg@mail.gmail.com>
References: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
	<CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>
	<CADOqKp=yAaK4s6cPVOgD_y_+tZTfXMGyKDXq8ad0gTE3WXFLyg@mail.gmail.com>
Message-ID: <CAMYG4GnWZc9r2U3QoZ4wvmSjXCUR_UiF0fOkFAdkL0d75KBLhQ@mail.gmail.com>

On Mon, May 4, 2015 at 5:41 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
wrote:

>
> Dear Matt,
>
> Actually the initial jacobian was identity. Regular SNES converges in 48
> iterations, GMRES in 19, NCG in 67,...
> Do you think SNESQN with the basiclineseach was the problem for divergence?
> If I use SNESQN by default should not it converge with initial identity
> jacobian?
>

Do you mean that you used an initial guess of the identity, or that the
Actual Jacobian was the identity at your
initial guess?

  Matt


> Best regards,
> Reza
>
>
>
>
> On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, May 4, 2015 at 2:11 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
>> wrote:
>>
>>>
>>> Dear PETSC representatives,
>>>
>>> I am solving a nonlinear problem with SNESNGMRES and it converges faster
>>> with less iterations compared to otehr SNES methods. Any idea why that is
>>> the case?
>>>
>>
>> It is impossible to tell with this information.
>>
>>
>>> Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for
>>> the linesearch option and nothing changes.
>>>
>>
>> This can happen, especially if your matrix is far from the identity.
>>
>>
>>> I presume it is using the default SNESQN method. Btw, there are three
>>> options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN"
>>> in teh manual. I tried to associate them with SNES however it seems these
>>> hyphened names don't work there. What am I missing?
>>>
>>
>> -snes_qn_scale_type <lbfgs,broyden,badbroyden>
>>
>> from
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetType.html
>>
>>    Thanks,
>>
>>     Matt
>>
>> Best regards,
>>> Ray
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/b2fbb12a/attachment.html>

From reza.yaghmaie2 at gmail.com  Mon May  4 17:57:33 2015
From: reza.yaghmaie2 at gmail.com (Reza Yaghmaie)
Date: Mon, 4 May 2015 18:57:33 -0400
Subject: [petsc-users] (no subject)
In-Reply-To: <CAMYG4GnWZc9r2U3QoZ4wvmSjXCUR_UiF0fOkFAdkL0d75KBLhQ@mail.gmail.com>
References: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
	<CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>
	<CADOqKp=yAaK4s6cPVOgD_y_+tZTfXMGyKDXq8ad0gTE3WXFLyg@mail.gmail.com>
	<CAMYG4GnWZc9r2U3QoZ4wvmSjXCUR_UiF0fOkFAdkL0d75KBLhQ@mail.gmail.com>
Message-ID: <CADOqKpk8vKAs8F2FxMnZRNVpAm5-YEau26kCSS4XDA1YZgLMcg@mail.gmail.com>

Dear Matt,

The initial guess was zero for all cases of SNES solvers. The initial
jacobian was identity for all cases. The system is small and is ran
sequentially.
I have to add that I use FDColoring routine for the jacobian as well.

Regards,
Ray





On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 4, 2015 at 5:41 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com
> <javascript:_e(%7B%7D,'cvml','reza.yaghmaie2 at gmail.com');>> wrote:
>
>>
>> Dear Matt,
>>
>> Actually the initial jacobian was identity. Regular SNES converges in 48
>> iterations, GMRES in 19, NCG in 67,...
>> Do you think SNESQN with the basiclineseach was the problem for
>> divergence?
>> If I use SNESQN by default should not it converge with initial identity
>> jacobian?
>>
>
> Do you mean that you used an initial guess of the identity, or that the
> Actual Jacobian was the identity at your
> initial guess?
>
>   Matt
>
>
>> Best regards,
>> Reza
>>
>>
>>
>>
>> On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com
>> <javascript:_e(%7B%7D,'cvml','knepley at gmail.com');>> wrote:
>>
>>> On Mon, May 4, 2015 at 2:11 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
>>> wrote:
>>>
>>>>
>>>> Dear PETSC representatives,
>>>>
>>>> I am solving a nonlinear problem with SNESNGMRES and it converges
>>>> faster with less iterations compared to otehr SNES methods. Any idea why
>>>> that is the case?
>>>>
>>>
>>> It is impossible to tell with this information.
>>>
>>>
>>>> Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for
>>>> the linesearch option and nothing changes.
>>>>
>>>
>>> This can happen, especially if your matrix is far from the identity.
>>>
>>>
>>>> I presume it is using the default SNESQN method. Btw, there are three
>>>> options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN"
>>>> in teh manual. I tried to associate them with SNES however it seems these
>>>> hyphened names don't work there. What am I missing?
>>>>
>>>
>>> -snes_qn_scale_type <lbfgs,broyden,badbroyden>
>>>
>>> from
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetType.html
>>>
>>>    Thanks,
>>>
>>>     Matt
>>>
>>> Best regards,
>>>> Ray
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/4d233845/attachment.html>

From knepley at gmail.com  Mon May  4 20:06:52 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 4 May 2015 20:06:52 -0500
Subject: [petsc-users] (no subject)
In-Reply-To: <CADOqKpk8vKAs8F2FxMnZRNVpAm5-YEau26kCSS4XDA1YZgLMcg@mail.gmail.com>
References: <CADOqKp=jqH1E8Oo+ythB3LybG5Q_x38_drPWtBVrWxEvLV-Otw@mail.gmail.com>
	<CAMYG4GmuZjvWBU4kwA4vV83gDTA8H+wRbY3Dvcd+pZWOQmbNXg@mail.gmail.com>
	<CADOqKp=yAaK4s6cPVOgD_y_+tZTfXMGyKDXq8ad0gTE3WXFLyg@mail.gmail.com>
	<CAMYG4GnWZc9r2U3QoZ4wvmSjXCUR_UiF0fOkFAdkL0d75KBLhQ@mail.gmail.com>
	<CADOqKpk8vKAs8F2FxMnZRNVpAm5-YEau26kCSS4XDA1YZgLMcg@mail.gmail.com>
Message-ID: <CAMYG4GkCFkekg=2MfcAoWbZ2A0+H6DwYgtw5jyiDf7K98=qCVQ@mail.gmail.com>

On Mon, May 4, 2015 at 5:57 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
wrote:

>
> Dear Matt,
>
> The initial guess was zero for all cases of SNES solvers. The initial
> jacobian was identity for all cases. The system is small and is ran
> sequentially.
> I have to add that I use FDColoring routine for the jacobian as well.
>

So, the initial Jacobian was the identity, but we have no idea how close
that is to the true Jacobian. You could use the
coloring routines (the default) combined with


http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetScaleType.html

using Jacobian, which sets the initial guess to the true Jacobian. That
would give us an idea whether the initial guess
impacted QN covergence.

  Thanks,

     Matt


> Regards,
> Ray
>
>
>
>
>
> On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, May 4, 2015 at 5:41 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com>
>> wrote:
>>
>>>
>>> Dear Matt,
>>>
>>> Actually the initial jacobian was identity. Regular SNES converges in 48
>>> iterations, GMRES in 19, NCG in 67,...
>>> Do you think SNESQN with the basiclineseach was the problem for
>>> divergence?
>>> If I use SNESQN by default should not it converge with initial identity
>>> jacobian?
>>>
>>
>> Do you mean that you used an initial guess of the identity, or that the
>> Actual Jacobian was the identity at your
>> initial guess?
>>
>>   Matt
>>
>>
>>> Best regards,
>>> Reza
>>>
>>>
>>>
>>>
>>> On Monday, May 4, 2015, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>>> On Mon, May 4, 2015 at 2:11 PM, Reza Yaghmaie <reza.yaghmaie2 at gmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>> Dear PETSC representatives,
>>>>>
>>>>> I am solving a nonlinear problem with SNESNGMRES and it converges
>>>>> faster with less iterations compared to otehr SNES methods. Any idea why
>>>>> that is the case?
>>>>>
>>>>
>>>> It is impossible to tell with this information.
>>>>
>>>>
>>>>> Also SNESQN diverges quickly. I tried to use SNESLINESEARCHBASIC for
>>>>> the linesearch option and nothing changes.
>>>>>
>>>>
>>>> This can happen, especially if your matrix is far from the identity.
>>>>
>>>>
>>>>> I presume it is using the default SNESQN method. Btw, there are three
>>>>> options for QN, as "SNES_QN_LBFGS, SNES_QN_BROYDEN, SNES_QN_BADBROYEN"
>>>>> in teh manual. I tried to associate them with SNES however it seems these
>>>>> hyphened names don't work there. What am I missing?
>>>>>
>>>>
>>>> -snes_qn_scale_type <lbfgs,broyden,badbroyden>
>>>>
>>>> from
>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESQNSetType.html
>>>>
>>>>    Thanks,
>>>>
>>>>     Matt
>>>>
>>>> Best regards,
>>>>> Ray
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150504/d74ff756/attachment.html>

From Lukasz.Kaczmarczyk at glasgow.ac.uk  Tue May  5 05:18:13 2015
From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk)
Date: Tue, 5 May 2015 10:18:13 +0000
Subject: [petsc-users] mac os x, problem with partitioning
Message-ID: <0AA8B80F-BC02-443B-8D49-6ED810076EE0@glasgow.ac.uk>

Hello,

I struggle with problem of matrix partitioning on McOSX for some time, since Maverick. Error is random, it runs ok for couple of times and then it beak. I using the newest version of petsc, for bitbucket repository, however this problem exist for long time for older versions too.

If I load partitioned mesh, f.e. using metis or any other partitioner code work without errors. Moreover, the same code I run on linux, redhat, ubutu, tested with valgrind and no problems like that ever existed. I used as well intel compiler on linux and following problems never existed. 

*) When I use ptscoch,

It is at the point of graph partitioning (random),

(2): ERROR: dgraphCheck: arc data do not match
[2]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
(3): ERROR: dgraphCheck: arc data do not match
[3]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
[3]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
[3]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
(0): ERROR: dgraphCheck: arc data do not match
[0]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
[0]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
[0]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
(1): ERROR: dgraphCheck: arc data do not match
[1]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
[1]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c

*) When I use parmetis

It is random once again, but problem appear later in the code, when matrix is assembled, 


[1]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
[1]PETSC ERROR: MoFEM version 0.2.1
[1]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
[1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
[1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
[0]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
[0]PETSC ERROR: MoFEM version 0.2.1
[0]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
[0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
[0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Inserting a new nonzero at global row/column (71, 354) into matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
[0]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
[0]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
[0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 616 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Argument out of range
[1]PETSC ERROR: Inserting a new nonzero at global row/column (300, 536) into matrix
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
[1]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
[1]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 613 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
#3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
[0]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[0]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[0]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
[0]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[0]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[0]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
[0]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[0]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: [1]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
[1]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[1]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[1]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
[1]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[1]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[1]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
[1]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
[1]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[1]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
[1]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
[1]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
[1]PETSC ERROR: #17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[1]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
[1]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
[1]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -ksp_atol 1e-10
[1]PETSC ERROR: -ksp_rtol 1e-10
[1]PETSC ERROR: -ksp_type fgmres
[1]PETSC ERROR: -mat_partitioning_type parmetis
[1]PETSC ERROR: -my_disp_order 3
[1]PETSC ERROR: -my_file LShape.cub
[1]PETSC ERROR: -my_max_post_proc_ref_level 0
[1]PETSC ERROR: -my_output_prt 1
[1]PETSC ERROR: -my_time_data_file data.txt
[1]PETSC ERROR: -my_vel_order 3
[1]PETSC ERROR: -pc_factor_mat_solver_package mumps
[1]PETSC ERROR: -pc_type lu
#17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[0]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 63.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -ksp_atol 1e-10
[0]PETSC ERROR: -ksp_rtol 1e-10
[0]PETSC ERROR: -ksp_type fgmres
[0]PETSC ERROR: -mat_partitioning_type parmetis
[0]PETSC ERROR: -my_disp_order 3
[0]PETSC ERROR: -my_file LShape.cub
[0]PETSC ERROR: -my_max_post_proc_ref_level 0
[0]PETSC ERROR: -my_output_prt 1
[0]PETSC ERROR: -my_time_data_file data.txt
[0]PETSC ERROR: -my_vel_order 3
[0]PETSC ERROR: -pc_factor_mat_solver_package mumps
[0]PETSC ERROR: -pc_type lu
[0]PETSC ERROR: -snes_atol 1e-7
[0]PETSC ERROR: -snes_linesearch_type basic
[0]PETSC ERROR: -snes_max_it 100
[0]PETSC ERROR: -snes_monitor
[0]PETSC ERROR: -snes_rtol 1e-7
[0]PETSC ERROR: -snes_type newtonls
[0]PETSC ERROR: -ts_dt 0.001
[0]PETSC ERROR: -ts_final_time 0.002
[0]PETSC ERROR: -ts_monitor
[0]PETSC ERROR: -ts_type alpha
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[0]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
[1]PETSC ERROR: -snes_atol 1e-7
[1]PETSC ERROR: -snes_linesearch_type basic
[1]PETSC ERROR: -snes_max_it 100
[1]PETSC ERROR: -snes_monitor
[1]PETSC ERROR: -snes_rtol 1e-7
[1]PETSC ERROR: -snes_type newtonls
[1]PETSC ERROR: -ts_dt 0.001
[1]PETSC ERROR: -ts_final_time 0.002
[1]PETSC ERROR: -ts_monitor
[1]PETSC ERROR: -ts_type alpha
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[2]PETSC ERROR: likely location of problem given in stack below
[2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[2]PETSC ERROR:       INSTEAD the line number of the start of the function
[2]PETSC ERROR:       is given.
[2]PETSC ERROR: [2] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
[2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[2]PETSC ERROR: [2] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
[2]PETSC ERROR: [2] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
[2]PETSC ERROR: [2] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[2]PETSC ERROR: [2] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[2]PETSC ERROR: [2] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
[2]PETSC ERROR: [2] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[2]PETSC ERROR: [2] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[2]PETSC ERROR: [2] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
[2]PETSC ERROR: [2] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
[2]PETSC ERROR: [2] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
[2]PETSC ERROR: [2] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[2]PETSC ERROR: [2] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
[2]PETSC ERROR: [2] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
[2]PETSC ERROR: [2] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
[2]PETSC ERROR: [2] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
[2]PETSC ERROR: [2] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
[2]PETSC ERROR: [2] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[2]PETSC ERROR: [2] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
[2]PETSC ERROR: [2] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
[2]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
[2]PETSC ERROR: MoFEM version 0.2.1
[2]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
[2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
[2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Signal received
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
[2]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
[2]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --do[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[3]PETSC ERROR: likely location of problem given in stack below
[3]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[3]PETSC ERROR:       INSTEAD the line number of the start of the function
[3]PETSC ERROR:       is given.
[3]PETSC ERROR: [3] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
[3]PETSC ERROR: [3] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
[3]PETSC ERROR: [3] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
[3]PETSC ERROR: [3] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
[3]PETSC ERROR: [3] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[3]PETSC ERROR: [3] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[3]PETSC ERROR: [3] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
[3]PETSC ERROR: [3] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[3]PETSC ERROR: [3] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
[3]PETSC ERROR: [3] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
[3]PETSC ERROR: [3] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
[3]PETSC ERROR: [3] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
[3]PETSC ERROR: [3] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[3]PETSC ERROR: [3] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
[3]PETSC ERROR: [3] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: [3] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: [3] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
[3]PETSC ERROR: [3] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: [3] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
[3]PETSC ERROR: [3] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
[3]PETSC ERROR: [3] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
[3]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
[3]PETSC ERROR: MoFEM version 0.2.1
[3]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
[3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
[3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Signal received
[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
[3]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
[3]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
[2]PETSC ERROR: #1 User provided function() line 0 in  unknown file
wnload-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
[3]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[dh23-181.eng.gla.ac.uk:06691] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[dh23-181.eng.gla.ac.uk:06691] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 4849528 bytes
Desc: configure.log
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/5047b03d/attachment-0001.obj>

From stefano.pedraglio at libero.it  Tue May  5 06:41:33 2015
From: stefano.pedraglio at libero.it (stefano.pedraglio at libero.it)
Date: Tue, 5 May 2015 13:41:33 +0200 (CEST)
Subject: [petsc-users] R: Re:  Load real matrix in complex PETSc build.
Message-ID: <1214634167.1685151430826093284.JavaMail.httpd@webmail-31.iol.local>

Hallo
Thank you for your fast answer. I think I did what you meant with this script.


from petsc4py import PETSc
import sys
import os

directory = os.path.join(os.environ['PETSC_DIR'], 'bin','pythonscripts')
sys.path.insert(0,directory)

import PetscBinaryIO

InputOutput = PetscBinaryIO.PetscBinaryIO()

K, = InputOutput.readBinaryFile('K.txt', mattype='sparse')
M, = InputOutput.readBinaryFile('M.txt', mattype='sparse')

InputOutput.complexscalars = True
InputOutput._update_dtypes()

InputOutput.writeBinaryFile('Kcomplex.txt', [K,])
InputOutput.writeBinaryFile('Mcomplex.txt', [M,])



When I try to calculate the eigensolutions with the complex matrices I find 
eigenpairs a little different from those I got with the real matrices, or at 
least a different part of the spectrum. In addition not all the eigenalues have 
their complex conjugate. I use the same parameters. 
problem type PGNHEP
shift 0
eps type EPSARPACK
spectral transform STSINVERT

Is this behaviour expected? Thank you very much.
Hallo
Stefano



>----Messaggio originale----
>Da: bsmith at mcs.anl.gov
>Data: 4-mag-2015 21.40
>A: "stefano.pedraglio at libero.it"<stefano.pedraglio at libero.it>
>Cc: <petsc-users at mcs.anl.gov>
>Ogg: Re: [petsc-users] Load real matrix in complex PETSc build.
>
>
>   You should be able to do it using the PetscBinaryIO.py tools we provide in 
the bin directory. Just read in the matrix as real in python and then write it 
out as complex using the methods there.
>
>   Barry
>
>> On May 4, 2015, at 2:34 PM, stefano.pedraglio at libero.it wrote:
>> 
>> Hallo
>> I have saved a large matrix using PETSc configured with scalar type real 
and I used binary viewer. I need now to compute the eigensolutions in another 
program using SLEPc and I need to use a complex shift. I have configured and 
compiled another PETSc and SLEPc build for complex scalars. I know it is not 
possible to load a matrix saved as real when using a program configured for 
complex numbers. Is there a workaround to perform this operation? Maybe saving 
and loading in some other format the matrix or using a  different library for 
eigenvalue computation. Thank you for any suggestion.
>> 
>> Hallo
>> Stefano
>
>



From bsmith at mcs.anl.gov  Tue May  5 07:28:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 5 May 2015 07:28:40 -0500
Subject: [petsc-users] mac os x, problem with partitioning
In-Reply-To: <0AA8B80F-BC02-443B-8D49-6ED810076EE0@glasgow.ac.uk>
References: <0AA8B80F-BC02-443B-8D49-6ED810076EE0@glasgow.ac.uk>
Message-ID: <A4B3A9D9-678C-4C0C-8969-B2BFF0D06966@mcs.anl.gov>


   We haven't see this before, I can't think of a particular reason why it would work under Linux and not MacOSX.  You said you ran with valgrind, was that only on Linux or have you run the cases that fail on Apple with valgrind on Apple? (Yes valgrind often isn't supported on Apple but the development version of valgrind now works for both maverick and Yosemite. I would most definitely recommend first running with valgrind on Apple if you have not. You can also try running with -start_in_debugger and when PTscotch crashes look at the values to see why it thinks "arc data do not match", it could be a bug that doesn't expose itself on Linux with the different compilers.

  The error with permetis 
> Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at global row/column (71, 354) into matrix
is not a crash like with ptscotch. Could it be that you are actually inserting new non-zero locations at later time steps ? Or does it get past this point on Linux with the exact same options. If you are inserting new nonzero locations and want to then you can use the MatSetOption() to set MAT_NEW_NONZERO_LOCATION_ERR to PETSC_FALSE

  Good luck,

   Barry



> On May 5, 2015, at 5:18 AM, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> 
> Hello,
> 
> I struggle with problem of matrix partitioning on McOSX for some time, since Maverick. Error is random, it runs ok for couple of times and then it beak. I using the newest version of petsc, for bitbucket repository, however this problem exist for long time for older versions too.
> 
> If I load partitioned mesh, f.e. using metis or any other partitioner code work without errors. Moreover, the same code I run on linux, redhat, ubutu, tested with valgrind and no problems like that ever existed. I used as well intel compiler on linux and following problems never existed. 
> 
> *) When I use ptscoch,
> 
> It is at the point of graph partitioning (random),
> 
> (2): ERROR: dgraphCheck: arc data do not match
> [2]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> (3): ERROR: dgraphCheck: arc data do not match
> [3]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [3]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> [3]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
> (0): ERROR: dgraphCheck: arc data do not match
> [0]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [0]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> [0]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
> (1): ERROR: dgraphCheck: arc data do not match
> [1]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [1]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> 
> *) When I use parmetis
> 
> It is random once again, but problem appear later in the code, when matrix is assembled, 
> 
> 
> [1]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [1]PETSC ERROR: MoFEM version 0.2.1
> [1]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [0]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [0]PETSC ERROR: MoFEM version 0.2.1
> [0]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at global row/column (71, 354) into matrix
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [0]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [0]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 616 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Argument out of range
> [1]PETSC ERROR: Inserting a new nonzero at global row/column (300, 536) into matrix
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [1]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [1]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 613 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
> #3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [0]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [0]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [0]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [0]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: [1]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [1]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [1]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [1]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [1]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [1]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: -ksp_atol 1e-10
> [1]PETSC ERROR: -ksp_rtol 1e-10
> [1]PETSC ERROR: -ksp_type fgmres
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -my_disp_order 3
> [1]PETSC ERROR: -my_file LShape.cub
> [1]PETSC ERROR: -my_max_post_proc_ref_level 0
> [1]PETSC ERROR: -my_output_prt 1
> [1]PETSC ERROR: -my_time_data_file data.txt
> [1]PETSC ERROR: -my_vel_order 3
> [1]PETSC ERROR: -pc_factor_mat_solver_package mumps
> [1]PETSC ERROR: -pc_type lu
> #17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [0]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
> with errorcode 63.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -ksp_atol 1e-10
> [0]PETSC ERROR: -ksp_rtol 1e-10
> [0]PETSC ERROR: -ksp_type fgmres
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -my_disp_order 3
> [0]PETSC ERROR: -my_file LShape.cub
> [0]PETSC ERROR: -my_max_post_proc_ref_level 0
> [0]PETSC ERROR: -my_output_prt 1
> [0]PETSC ERROR: -my_time_data_file data.txt
> [0]PETSC ERROR: -my_vel_order 3
> [0]PETSC ERROR: -pc_factor_mat_solver_package mumps
> [0]PETSC ERROR: -pc_type lu
> [0]PETSC ERROR: -snes_atol 1e-7
> [0]PETSC ERROR: -snes_linesearch_type basic
> [0]PETSC ERROR: -snes_max_it 100
> [0]PETSC ERROR: -snes_monitor
> [0]PETSC ERROR: -snes_rtol 1e-7
> [0]PETSC ERROR: -snes_type newtonls
> [0]PETSC ERROR: -ts_dt 0.001
> [0]PETSC ERROR: -ts_final_time 0.002
> [0]PETSC ERROR: -ts_monitor
> [0]PETSC ERROR: -ts_type alpha
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [0]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
> [1]PETSC ERROR: -snes_atol 1e-7
> [1]PETSC ERROR: -snes_linesearch_type basic
> [1]PETSC ERROR: -snes_max_it 100
> [1]PETSC ERROR: -snes_monitor
> [1]PETSC ERROR: -snes_rtol 1e-7
> [1]PETSC ERROR: -snes_type newtonls
> [1]PETSC ERROR: -ts_dt 0.001
> [1]PETSC ERROR: -ts_final_time 0.002
> [1]PETSC ERROR: -ts_monitor
> [1]PETSC ERROR: -ts_type alpha
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [2]PETSC ERROR:       INSTEAD the line number of the start of the function
> [2]PETSC ERROR:       is given.
> [2]PETSC ERROR: [2] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
> [2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [2]PETSC ERROR: [2] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
> [2]PETSC ERROR: [2] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [2]PETSC ERROR: [2] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [2]PETSC ERROR: [2] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [2]PETSC ERROR: [2] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [2]PETSC ERROR: [2] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
> [2]PETSC ERROR: [2] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [2]PETSC ERROR: [2] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [2]PETSC ERROR: MoFEM version 0.2.1
> [2]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: Signal received
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [2]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [2]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [2]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --do[3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [3]PETSC ERROR: likely location of problem given in stack below
> [3]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [3]PETSC ERROR:       INSTEAD the line number of the start of the function
> [3]PETSC ERROR:       is given.
> [3]PETSC ERROR: [3] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
> [3]PETSC ERROR: [3] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [3]PETSC ERROR: [3] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
> [3]PETSC ERROR: [3] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [3]PETSC ERROR: [3] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [3]PETSC ERROR: [3] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [3]PETSC ERROR: [3] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [3]PETSC ERROR: [3] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
> [3]PETSC ERROR: [3] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [3]PETSC ERROR: [3] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [3]PETSC ERROR: MoFEM version 0.2.1
> [3]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [3]PETSC ERROR: Signal received
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [3]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [3]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [3]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [2]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> wnload-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [3]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [dh23-181.eng.gla.ac.uk:06691] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
> [dh23-181.eng.gla.ac.uk:06691] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> <configure.log>


From simpson at math.drexel.edu  Tue May  5 12:12:59 2015
From: simpson at math.drexel.edu (Gideon Simpson)
Date: Tue, 5 May 2015 13:12:59 -0400
Subject: [petsc-users] TSMonitorSet question
Message-ID: <CA0DA1F9-0B0E-4CC0-A649-DDF1784C0D32@math.drexel.edu>

Is TSMonitorSet called before or after the new time step is taken (and presumably accepted)?

-gideon


From jed at jedbrown.org  Tue May  5 14:16:30 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 May 2015 15:16:30 -0400
Subject: [petsc-users] TSMonitorSet question
In-Reply-To: <CA0DA1F9-0B0E-4CC0-A649-DDF1784C0D32@math.drexel.edu>
References: <CA0DA1F9-0B0E-4CC0-A649-DDF1784C0D32@math.drexel.edu>
Message-ID: <878ud2kf75.fsf@jedbrown.org>

Gideon Simpson <simpson at math.drexel.edu> writes:

> Is TSMonitorSet called before or after the new time step is taken (and presumably accepted)?

I assume you mean TSMonitor.  From TSSolve:

    while (!ts->reason) {
      ierr = TSMonitor(ts,ts->steps,ts->ptime,ts->vec_sol);CHKERRQ(ierr);
      ierr = TSStep(ts);CHKERRQ(ierr);
      if (ts->event) {
	ierr = TSEventMonitor(ts);CHKERRQ(ierr);
      }
      if(!ts->steprollback) {
	ierr = TSTrajectorySet(ts->trajectory,ts,ts->steps,ts->ptime,ts->vec_sol);CHKERRQ(ierr);
	ierr = TSPostStep(ts);CHKERRQ(ierr);
      }
    }
    if (ts->exact_final_time == TS_EXACTFINALTIME_INTERPOLATE && ts->ptime > ts->max_time) {
      ierr = TSInterpolate(ts,ts->max_time,u);CHKERRQ(ierr);
      ts->solvetime = ts->max_time;
      solution = u;
    } else {
      if (u) {ierr = VecCopy(ts->vec_sol,u);CHKERRQ(ierr);}
      ts->solvetime = ts->ptime;
      solution = ts->vec_sol;
    }
    ierr = TSMonitor(ts,ts->steps,ts->solvetime,solution);CHKERRQ(ierr);
    ierr = VecViewFromOptions(solution, ((PetscObject) ts)->prefix, "-ts_view_solution");CHKERRQ(ierr);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/ead28a46/attachment.pgp>

From bikash at umich.edu  Tue May  5 15:34:04 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Tue, 5 May 2015 16:34:04 -0400
Subject: [petsc-users] Global matrix from global sub-matrices
Message-ID: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>

Hi,

I have four global matrices: A (mxm), B (mxn), C (nxm) and D (nxn). All of
these are of type MATMPIAIJ (sparse parallel matrix) created using number
of global rows and columns. I want to construct an (m+n)x(m+n) global
matrix called M which would have A, B, C and D as its sub-matrices in
clockwise fashion starting from A at the upper-left. What can be the best
way to construct M from A, B, C and D sub-matrices?

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/19717900/attachment.html>

From knepley at gmail.com  Tue May  5 15:34:53 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 May 2015 15:34:53 -0500
Subject: [petsc-users] advection-diffusion pointwise function for FEM
In-Reply-To: <CAP2=TMgQ7bF8N=6nhwk2i5=ZNNrpVR-+7fWN7Kmyj2mDHGUxPA@mail.gmail.com>
References: <CAP2=TMgQ7bF8N=6nhwk2i5=ZNNrpVR-+7fWN7Kmyj2mDHGUxPA@mail.gmail.com>
Message-ID: <CAMYG4GnULoUjZYRoamfCUOkXsi6Bk-85o7BBy4W3kf4Fevq70w@mail.gmail.com>

On Sat, Apr 4, 2015 at 11:54 AM, Justin Chang <jchang27 at uh.edu> wrote:

> Hi everyone,
>
> I want to include advection into my diffusion FEM code (I am assuming a
> small Peclet number so stability isn't an issue ATM). That is I want to
> incorporate the second term as a pointwise function:
>
> du/dt + v * grad[c] - div[grad[c]] = f
>
> Where v is the velocity (obtained from the auxiliary term a_x[]). For the
> residual, would it be of the following form:
>

1) I would think you would use a[] instead. What is your velocity the
gradient of?


> f0_u(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar
> u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
> a_x[], const PetscReal x[], PetscScalar f0[]) {
>
>   PetscInt d;
>   f[0] = u_t[0];
>   for (d = 0; d < spatialDim; ++d) f0[0] += a_x[d]*u_x[d];
>
> }
>
> What about the jacobian? My guess would be to use g1(...) but what would
> the inside of this function be?
>

Yes it would be g1. The indices for the output are f,g,dg. I am guessing
that c is a scalar, so f = {0}, g = {0}, dg = {0, 1}
for 2D, so g1 would have two terms,

  g1[0] = a[0];
  g1[1] = a[1];

  Thanks,

    Matt


> Thanks,
>
>
> --
> Justin Chang
> PhD Candidate, Civil Engineering - Computational Sciences
> University of Houston, Department of Civil and Environmental Engineering
> Houston, TX 77004
> (512) 963-3262
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/12090388/attachment.html>

From jed at jedbrown.org  Tue May  5 15:38:58 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 05 May 2015 16:38:58 -0400
Subject: [petsc-users] Global matrix from global sub-matrices
In-Reply-To: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
References: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
Message-ID: <87383akbdp.fsf@jedbrown.org>

Bikash Kanungo <bikash at umich.edu> writes:

> Hi,
>
> I have four global matrices: A (mxm), B (mxn), C (nxm) and D (nxn). All of
> these are of type MATMPIAIJ (sparse parallel matrix) created using number
> of global rows and columns. I want to construct an (m+n)x(m+n) global
> matrix called M which would have A, B, C and D as its sub-matrices in
> clockwise fashion starting from A at the upper-left. What can be the best
> way to construct M from A, B, C and D sub-matrices?

You can build it with MatNest, but that format is constraining so unless
you understand the tradeoffs for different preconditioning techniques
applied to your specific problem, you should instead allocate the big
matrix, use MatGetLocalSubMatrix() to get references to the logical
blocks, and call your functions that assemble each of A,B,C,D.  See
src/snes/examples/tutorials/ex28.c for an example of this organization.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/aed00bf5/attachment.pgp>

From xzhao99 at gmail.com  Tue May  5 15:40:07 2015
From: xzhao99 at gmail.com (Xujun Zhao)
Date: Tue, 5 May 2015 15:40:07 -0500
Subject: [petsc-users] Global matrix from global sub-matrices
In-Reply-To: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
References: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
Message-ID: <CAHOKZ65GEGwkVJnBBwRV52R59oiO8aaoJde+zHCVO8qwgJAH6g@mail.gmail.com>

Hi Bikash,

I think you can use nested matrix. See MatCreateNest and the example
src/snes/examples/tutorials/ex70.c.html
<http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex70.c.html>

Xujun

On Tue, May 5, 2015 at 3:34 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I have four global matrices: A (mxm), B (mxn), C (nxm) and D (nxn). All of
> these are of type MATMPIAIJ (sparse parallel matrix) created using number
> of global rows and columns. I want to construct an (m+n)x(m+n) global
> matrix called M which would have A, B, C and D as its sub-matrices in
> clockwise fashion starting from A at the upper-left. What can be the best
> way to construct M from A, B, C and D sub-matrices?
>
> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/ed42d1b0/attachment.html>

From knepley at gmail.com  Tue May  5 15:40:46 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 5 May 2015 15:40:46 -0500
Subject: [petsc-users] Global matrix from global sub-matrices
In-Reply-To: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
References: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
Message-ID: <CAMYG4G=XpXzCGdZEZBaAoEGUwJ8-2=zA26JKojbx0Ja8XekYfA@mail.gmail.com>

On Tue, May 5, 2015 at 3:34 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I have four global matrices: A (mxm), B (mxn), C (nxm) and D (nxn). All of
> these are of type MATMPIAIJ (sparse parallel matrix) created using number
> of global rows and columns. I want to construct an (m+n)x(m+n) global
> matrix called M which would have A, B, C and D as its sub-matrices in
> clockwise fashion starting from A at the upper-left. What can be the best
> way to construct M from A, B, C and D sub-matrices?
>

There are at least two ways:

  1) Use
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html
to get a view
      into the matrix for each submatrix A, B, C, D

  2) Use
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html

  Thanks,

     Matt


> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/70defe65/attachment-0001.html>

From bikash at umich.edu  Tue May  5 16:45:48 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Tue, 5 May 2015 17:45:48 -0400
Subject: [petsc-users] Global matrix from global sub-matrices
In-Reply-To: <CAMYG4G=XpXzCGdZEZBaAoEGUwJ8-2=zA26JKojbx0Ja8XekYfA@mail.gmail.com>
References: <CA+H+vqLnGkys0K0XhJ9xyjqgCk_zsdEqNR25u6rfRvdF+VgEyg@mail.gmail.com>
	<CAMYG4G=XpXzCGdZEZBaAoEGUwJ8-2=zA26JKojbx0Ja8XekYfA@mail.gmail.com>
Message-ID: <CA+H+vqLX0_uBcBoCbxyWxMutHQiCFa25cLe921F_v6sjW+-gvQ@mail.gmail.com>

Thank you guys for the prompt replies.

I have never used DMs so far in my research. On a cursory glance
MatCreateNest seems to be simpler. But I would like to dig into
MatGelLocalSubmatrix to see if it provides any advantage.

Regards,
Bikash
On Tue, May 5, 2015 at 3:34 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I have four global matrices: A (mxm), B (mxn), C (nxm) and D (nxn). All of
> these are of type MATMPIAIJ (sparse parallel matrix) created using number
> of global rows and columns. I want to construct an (m+n)x(m+n) global
> matrix called M which would have A, B, C and D as its sub-matrices in
> clockwise fashion starting from A at the upper-left. What can be the best
> way to construct M from A, B, C and D sub-matrices?
>

There are at least two ways:

  1) Use
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html
to get a view
      into the matrix for each submatrix A, B, C, D

  2) Use
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateNest.html

  Thanks,

     Matt


> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150505/ced4729c/attachment.html>

From jychang48 at gmail.com  Wed May  6 02:07:52 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 02:07:52 -0500
Subject: [petsc-users] advection-diffusion pointwise function for FEM
In-Reply-To: <CAMYG4GnULoUjZYRoamfCUOkXsi6Bk-85o7BBy4W3kf4Fevq70w@mail.gmail.com>
References: <CAP2=TMgQ7bF8N=6nhwk2i5=ZNNrpVR-+7fWN7Kmyj2mDHGUxPA@mail.gmail.com>
	<CAMYG4GnULoUjZYRoamfCUOkXsi6Bk-85o7BBy4W3kf4Fevq70w@mail.gmail.com>
Message-ID: <CAP2=TMjqRDE1nBE_Z45+difnnBm=TdYanJ5ibkW0AVRunaz0rw@mail.gmail.com>

Matt,

Yes I would actually use a[] in this case. Back then, I think I was trying
to take the derivative of the auxiliary which i had declared to be
piece-wise constant.

The auxiliary a[] is the gradient of pressure (times permeability over
viscosity) obtained from solving the Darcy equation as a Laplacian. This
part is solved separately, projected as cell-wise velocity, and inputted as
an auxiliary for the advection-diffusion problem.

Speaking of which, have you had a chance to look into
DMPlexProjectFieldLocal(...) not projecting dirichlet BC constraints? I
realize that the "small example" I gave you wasn't small at all, so I can
send you a smaller and simpler one if needed.

Thanks,
Justin

On Tue, May 5, 2015 at 3:34 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Apr 4, 2015 at 11:54 AM, Justin Chang <jchang27 at uh.edu> wrote:
>
>> Hi everyone,
>>
>> I want to include advection into my diffusion FEM code (I am assuming a
>> small Peclet number so stability isn't an issue ATM). That is I want to
>> incorporate the second term as a pointwise function:
>>
>> du/dt + v * grad[c] - div[grad[c]] = f
>>
>> Where v is the velocity (obtained from the auxiliary term a_x[]). For the
>> residual, would it be of the following form:
>>
>
> 1) I would think you would use a[] instead. What is your velocity the
> gradient of?
>
>
>> f0_u(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar
>> u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
>> a_x[], const PetscReal x[], PetscScalar f0[]) {
>>
>>   PetscInt d;
>>   f[0] = u_t[0];
>>   for (d = 0; d < spatialDim; ++d) f0[0] += a_x[d]*u_x[d];
>>
>> }
>>
>> What about the jacobian? My guess would be to use g1(...) but what would
>> the inside of this function be?
>>
>
> Yes it would be g1. The indices for the output are f,g,dg. I am guessing
> that c is a scalar, so f = {0}, g = {0}, dg = {0, 1}
> for 2D, so g1 would have two terms,
>
>   g1[0] = a[0];
>   g1[1] = a[1];
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>>
>>
>> --
>> Justin Chang
>> PhD Candidate, Civil Engineering - Computational Sciences
>> University of Houston, Department of Civil and Environmental Engineering
>> Houston, TX 77004
>> (512) 963-3262
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/ff4e8e45/attachment.html>

From jychang48 at gmail.com  Wed May  6 02:30:27 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 02:30:27 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87bni0jph7.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
Message-ID: <CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>

Thank you guys for your responses. If I want to estimate the number of
bytes that come down, would -memory_info give me that information?

And with this information plus the total number of logged flops, i can get
the ratio of flop to bytes and hence the (crude estimation of) upper bound
FLOPS/s based on the reported stream BW?

Thanks,
Justin

On Mon, May 4, 2015 at 11:07 AM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > Hi Jed,
> >
> > Thanks for the reply. Not too long ago one of you guys (Matt I think) had
> > mentioned the Roofline model and I was hoping to emulate something like
> it
> > for my application. If I understand the presentation slides (and the
> paper
> > implementing it) correctly, the upper bound FLOPS/s is calculated by
> > multiplying the stream BW by the ratio of DRAM flop to byte (aka
> arithmetic
> > intensity). The workload (i.e., flops) can be counted via PetscLogFlops()
> > and in the paper, the sparse matvec total bytes transferred for fmadd was
> > manually counted. Since my program involves more than just matvec I am
> > curious if there's a way to obtain the bytes for all operations and
> > functions invoked.
>
> Counting "useful" data motion subject to some cache granularity is not
> automatic.  You can look at performance analysis of stencil operations
> for an example of what this can look like.  I go through examples in my
> class, but I do it interactively with experiments rather than off of
> slides.
>
> > Or if I really should go with what you had suggested, could you
> elaborate a
> > little more on it, or point me to some papers/links/slides that talk
> about
> > it?
> >
> > Thanks,
> > Justin
> >
> > On Monday, May 4, 2015, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Justin Chang <jychang48 at gmail.com> writes:
> >>
> >> > Hello everyone,
> >> >
> >> > If I wanted to obtain the bytes/second for my PETSc program, is there
> a
> >> > generic way of doing this? My initial thought would be to first run
> the
> >> > program with valgrind to obtain the total memory usage, and then run
> it
> >> > without valgrind to get the wall clock time. These two metrics then
> give
> >> > you the bytes/second.
> >>
> >> Not really, because usually we're interested in useful bandwidth
> >> sustained from some level of cache.  You can use hardware performance
> >> counters to measure the number of cache lines transferred, but this is
> >> usually an overestimate of the amount of useful data.  You really need a
> >> performance model for your application and a cache model for the machine
> >> to say what bandwidth is useful.
> >>
> >> > Or can PETSc manually count the load/stores the way it's done for
> >> > flops?
> >>
> >> No, this information is not available in source code and would be nearly
> >> meaningless even if it was.
> >>
> >> > I was looking at the PetscMemXXX() functions but wasn't sure if this
> >> > is what I was looking for.
> >> >
> >> > Thanks,
> >> > Justin
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/e5a6e70d/attachment.html>

From E.Mueller at bath.ac.uk  Wed May  6 03:14:48 2015
From: E.Mueller at bath.ac.uk (Eike Mueller)
Date: Wed, 6 May 2015 08:14:48 +0000
Subject: [petsc-users] Seting advanced smoothers for BoomerAMG through PETSc
Message-ID: <1E962B7A-BE00-43E3-A202-37E64BD58371@bath.ac.uk>

Dear PETSc,

I?m trying to use hypre?s BoomerAMG preconditioner through PETSc. However, I would like to use a non-standard smoother, namely an incomplete LU factorisation, which is implemented in hypre?s Euclid (that?s because the problem is highly anisotropic, and GAMG worked very weill with ILU smoother). In native hypre I can set this with HYPRE_BoomerAMGSetSmoothType(). However, I can?t see how I can pass this through the PETSc interface, since the only boomeramg options I can set are listed below. As far as I can see they allow me to change the pointwise smoother via HYPRE_BoomerAMGSetRelaxType().

Does anyone have any advice on what I can do to use Euclid as a smoother via setting the correct PETSc options?

Thank you very much,

Eike

eikemueller at 138-38-173-198 $ ./ex6 -f -f helmholtz-sphere.dat  -ksp_type cg -ksp_convergence_test skip -ksp_max_it 2  -ksp_monitor -table -pc_hypre_type boomeramg -pc_type hypre -log_summary -ksp_view -help | grep boomeramg
  -pc_hypre_type <boomeramg> (choose one of) pilut parasails boomeramg ams (PCHYPRESetType)
  -pc_hypre_boomeramg_cycle_type <V> (choose one of) V W (None)
  -pc_hypre_boomeramg_max_levels <25>: Number of levels (of grids) allowed (None)
  -pc_hypre_boomeramg_max_iter <1>: Maximum iterations used PER hypre call (None)
  -pc_hypre_boomeramg_tol <0>: Convergence tolerance PER hypre call (0.0 = use a fixed number of iterations) (None)
  -pc_hypre_boomeramg_truncfactor <0>: Truncation factor for interpolation (0=no truncation) (None)
  -pc_hypre_boomeramg_P_max <0>: Max elements per row for interpolation operator (0=unlimited) (None)
  -pc_hypre_boomeramg_agg_nl <0>: Number of levels of aggressive coarsening (None)
  -pc_hypre_boomeramg_agg_num_paths <1>: Number of paths for aggressive coarsening (None)
  -pc_hypre_boomeramg_strong_threshold <0.25>: Threshold for being strongly connected (None)
  -pc_hypre_boomeramg_max_row_sum <0.9>: Maximum row sum (None)
  -pc_hypre_boomeramg_grid_sweeps_all <1>: Number of sweeps for the up and down grid levels (None)
  -pc_hypre_boomeramg_grid_sweeps_down <1>: Number of sweeps for the down cycles (None)
  -pc_hypre_boomeramg_grid_sweeps_up <1>: Number of sweeps for the up cycles (None)
  -pc_hypre_boomeramg_grid_sweeps_coarse <1>: Number of sweeps for the coarse level (None)
  -pc_hypre_boomeramg_relax_type_all <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
  -pc_hypre_boomeramg_relax_type_down <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
  -pc_hypre_boomeramg_relax_type_up <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
  -pc_hypre_boomeramg_relax_type_coarse <Gaussian-elimination> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
  -pc_hypre_boomeramg_relax_weight_all <1>: Relaxation weight for all levels (0 = hypre estimates, -k = determined with k CG steps) (None)
  -pc_hypre_boomeramg_relax_weight_level <1>: Set the relaxation weight for a particular level (weight,level) (None)
  -pc_hypre_boomeramg_outer_relax_weight_all <1>: Outer relaxation weight for all levels (-k = determined with k CG steps) (None)
  -pc_hypre_boomeramg_outer_relax_weight_level <1>: Set the outer relaxation weight for a particular level (weight,level) (None)
  -pc_hypre_boomeramg_no_CF: <FALSE> Do not use CF-relaxation (None)
  -pc_hypre_boomeramg_measure_type <local> (choose one of) local global (None)
  -pc_hypre_boomeramg_coarsen_type <Falgout> (choose one of) CLJP Ruge-Stueben  modifiedRuge-Stueben   Falgout  PMIS  HMIS (None)
  -pc_hypre_boomeramg_interp_type <classical> (choose one of) classical   direct multipass multipass-wts ext+i ext+i-cc standard standard-wts   FF FF1 (None)
  -pc_hypre_boomeramg_print_statistics: Print statistics (None)
  -pc_hypre_boomeramg_print_debug: Print debug information (None)
  -pc_hypre_boomeramg_nodal_coarsen: <FALSE> HYPRE_BoomerAMGSetNodal() (None)
  -pc_hypre_boomeramg_nodal_relaxation: <FALSE> Nodal relaxation via Schwarz (None)
-pc_hypre_type boomeramg

--

Dr Eike Hermann Mueller
Lecturer in Scientific Computing

Department of Mathematical Sciences
University of Bath
Bath BA2 7AY, United Kingdom

+44 1225 38 6241
e.mueller at bath.ac.uk<mailto:e.mueller at bath.ac.uk>
http://people.bath.ac.uk/em459/





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/85670673/attachment-0001.html>

From siddhesh4godbole at gmail.com  Wed May  6 04:16:24 2015
From: siddhesh4godbole at gmail.com (siddhesh godbole)
Date: Wed, 6 May 2015 14:46:24 +0530
Subject: [petsc-users] PetscFinalize in the middle of the C program
Message-ID: <CAO3zjjnDJ3H=_ZSt6NMDFjt4PwHS6X8cdc=B2J+g8+XqTWvmgQ@mail.gmail.com>

Hello,

I want to close the PETSc environment in the middle of my C program but
want to retain the matrices generated in PETSc and access them using C
program. How can i /(Can I) do this?

To be precise, i am solving eigenvalue system using SLEPc and i am using
the results to decouple the system matrices. Now i want to run the Newmark
integration code on those matrices. I am getting very slow performance by
PETSc routines such as
MatGetColumnVector()
VecGetArray()
MatAXPY()
VecAXPBYPCZ()
 So i just want to close the PETSc environment and retain these matrices
and solve using available Newmark code in C using mpi.  Need your inputs.

Thank you very much

*Siddhesh M Godbole*

5th year Dual Degree,
Civil Eng & Applied Mech.
IIT Madras
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/b80af4e7/attachment.html>

From jed at jedbrown.org  Wed May  6 06:43:41 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 06 May 2015 07:43:41 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
Message-ID: <87twvqgccy.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Thank you guys for your responses. If I want to estimate the number of
> bytes that come down, would -memory_info give me that information?

No.  There is no substitute for a performance model.  Hardware
performance counters can tell you something, but without a cache-aware
performance model, it can give you information that is off by an order
of magnitude or more.

> And with this information plus the total number of logged flops, i can get
> the ratio of flop to bytes and hence the (crude estimation of) upper bound
> FLOPS/s based on the reported stream BW?

Flop/s is logged (for PETSc operations) and you can use hardware
performance counters for that (much better than for bandwidth, but still
erroneous due to speculative execution and incomplete/gratuitous
vectorization).  But to compute an arithmetic intensity or find
performance bounds based on memory bandwidth considerations, there is no
substitute for understanding what you are doing.  Sorry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/3a10c55f/attachment.pgp>

From bsmith at mcs.anl.gov  Wed May  6 07:27:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 6 May 2015 07:27:23 -0500
Subject: [petsc-users] Seting advanced smoothers for BoomerAMG through
	PETSc
In-Reply-To: <1E962B7A-BE00-43E3-A202-37E64BD58371@bath.ac.uk>
References: <1E962B7A-BE00-43E3-A202-37E64BD58371@bath.ac.uk>
Message-ID: <14B9A4F8-5114-4579-BF5B-BBDFE265B1C8@mcs.anl.gov>

  Eike,

   You'll need to edit src/ksp/pc/impls/hypre/hypre.c and add an option to set this (where you see all the other BoomerAMG options). Then run make in
the root PETSc directory.

   If you do this in the development copy of PETSc and make a pull request (see https://bitbucket.org/petsc/petsc/wiki/Home Contributing to PETSc) then all future users can benefit from your additions.

  Barry

> On May 6, 2015, at 3:14 AM, Eike Mueller <E.Mueller at bath.ac.uk> wrote:
> 
> Dear PETSc,
> 
> I?m trying to use hypre?s BoomerAMG preconditioner through PETSc. However, I would like to use a non-standard smoother, namely an incomplete LU factorisation, which is implemented in hypre?s Euclid (that?s because the problem is highly anisotropic, and GAMG worked very weill with ILU smoother). In native hypre I can set this with HYPRE_BoomerAMGSetSmoothType(). However, I can?t see how I can pass this through the PETSc interface, since the only boomeramg options I can set are listed below. As far as I can see they allow me to change the pointwise smoother via HYPRE_BoomerAMGSetRelaxType().
> 
> Does anyone have any advice on what I can do to use Euclid as a smoother via setting the correct PETSc options?
> 
> Thank you very much,
> 
> Eike
> 
> eikemueller at 138-38-173-198 $ ./ex6 -f -f helmholtz-sphere.dat  -ksp_type cg -ksp_convergence_test skip -ksp_max_it 2  -ksp_monitor -table -pc_hypre_type boomeramg -pc_type hypre -log_summary -ksp_view -help | grep boomeramg
>   -pc_hypre_type <boomeramg> (choose one of) pilut parasails boomeramg ams (PCHYPRESetType)
>   -pc_hypre_boomeramg_cycle_type <V> (choose one of) V W (None)
>   -pc_hypre_boomeramg_max_levels <25>: Number of levels (of grids) allowed (None)
>   -pc_hypre_boomeramg_max_iter <1>: Maximum iterations used PER hypre call (None)
>   -pc_hypre_boomeramg_tol <0>: Convergence tolerance PER hypre call (0.0 = use a fixed number of iterations) (None)
>   -pc_hypre_boomeramg_truncfactor <0>: Truncation factor for interpolation (0=no truncation) (None)
>   -pc_hypre_boomeramg_P_max <0>: Max elements per row for interpolation operator (0=unlimited) (None)
>   -pc_hypre_boomeramg_agg_nl <0>: Number of levels of aggressive coarsening (None)
>   -pc_hypre_boomeramg_agg_num_paths <1>: Number of paths for aggressive coarsening (None)
>   -pc_hypre_boomeramg_strong_threshold <0.25>: Threshold for being strongly connected (None)
>   -pc_hypre_boomeramg_max_row_sum <0.9>: Maximum row sum (None)
>   -pc_hypre_boomeramg_grid_sweeps_all <1>: Number of sweeps for the up and down grid levels (None)
>   -pc_hypre_boomeramg_grid_sweeps_down <1>: Number of sweeps for the down cycles (None)
>   -pc_hypre_boomeramg_grid_sweeps_up <1>: Number of sweeps for the up cycles (None)
>   -pc_hypre_boomeramg_grid_sweeps_coarse <1>: Number of sweeps for the coarse level (None)
>   -pc_hypre_boomeramg_relax_type_all <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>   -pc_hypre_boomeramg_relax_type_down <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>   -pc_hypre_boomeramg_relax_type_up <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>   -pc_hypre_boomeramg_relax_type_coarse <Gaussian-elimination> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>   -pc_hypre_boomeramg_relax_weight_all <1>: Relaxation weight for all levels (0 = hypre estimates, -k = determined with k CG steps) (None)
>   -pc_hypre_boomeramg_relax_weight_level <1>: Set the relaxation weight for a particular level (weight,level) (None)
>   -pc_hypre_boomeramg_outer_relax_weight_all <1>: Outer relaxation weight for all levels (-k = determined with k CG steps) (None)
>   -pc_hypre_boomeramg_outer_relax_weight_level <1>: Set the outer relaxation weight for a particular level (weight,level) (None)
>   -pc_hypre_boomeramg_no_CF: <FALSE> Do not use CF-relaxation (None)
>   -pc_hypre_boomeramg_measure_type <local> (choose one of) local global (None)
>   -pc_hypre_boomeramg_coarsen_type <Falgout> (choose one of) CLJP Ruge-Stueben  modifiedRuge-Stueben   Falgout  PMIS  HMIS (None)
>   -pc_hypre_boomeramg_interp_type <classical> (choose one of) classical   direct multipass multipass-wts ext+i ext+i-cc standard standard-wts   FF FF1 (None)
>   -pc_hypre_boomeramg_print_statistics: Print statistics (None)
>   -pc_hypre_boomeramg_print_debug: Print debug information (None)
>   -pc_hypre_boomeramg_nodal_coarsen: <FALSE> HYPRE_BoomerAMGSetNodal() (None)
>   -pc_hypre_boomeramg_nodal_relaxation: <FALSE> Nodal relaxation via Schwarz (None)
> -pc_hypre_type boomeramg
> 
> --
> 
> Dr Eike Hermann Mueller
> Lecturer in Scientific Computing
> 
> Department of Mathematical Sciences
> University of Bath
> Bath BA2 7AY, United Kingdom
> 
> +44 1225 38 6241
> e.mueller at bath.ac.uk
> http://people.bath.ac.uk/em459/
> 
> 
> 
> 
> 


From bsmith at mcs.anl.gov  Wed May  6 07:31:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 6 May 2015 07:31:48 -0500
Subject: [petsc-users] PetscFinalize in the middle of the C program
In-Reply-To: <CAO3zjjnDJ3H=_ZSt6NMDFjt4PwHS6X8cdc=B2J+g8+XqTWvmgQ@mail.gmail.com>
References: <CAO3zjjnDJ3H=_ZSt6NMDFjt4PwHS6X8cdc=B2J+g8+XqTWvmgQ@mail.gmail.com>
Message-ID: <E7E34BF7-8F37-4C95-AA37-11A6303745B5@mcs.anl.gov>


> On May 6, 2015, at 4:16 AM, siddhesh godbole <siddhesh4godbole at gmail.com> wrote:
> 
> Hello,
> 
> I want to close the PETSc environment in the middle of my C program but want to retain the matrices generated in PETSc and access them using C program. How can i /(Can I) do this? 
> 
> To be precise, i am solving eigenvalue system using SLEPc and i am using the results to decouple the system matrices. Now i want to run the Newmark integration code on those matrices. I am getting very slow performance by PETSc routines such as
> MatGetColumnVector()
> VecGetArray()
> MatAXPY()
> VecAXPBYPCZ()
>  So i just want to close the PETSc environment and retain these matrices and solve using available Newmark code in C using mpi.  Need your inputs.
> 
   You cannot call PetscFinalize() and then continue to use PETSc matrices; but you can avoid using routines that are slow for you. Now MatGetColumnVector() and MatAXPY() can be slow depending on the matrices nonzero structure but VecGetArray() and VecAXPBYPCZ() will never be slow. Perhaps you need to look at the timings more carefully to see where the time is being spent. For example run with -log_summary and email the results to us so we can determine what is causing your slow performance.

  Barry

   
> Thank you very much
> 
> Siddhesh M Godbole
> 
> 5th year Dual Degree,
> Civil Eng & Applied Mech.
> IIT Madras


From knepley at gmail.com  Wed May  6 08:06:00 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 08:06:00 -0500
Subject: [petsc-users] advection-diffusion pointwise function for FEM
In-Reply-To: <CAP2=TMjqRDE1nBE_Z45+difnnBm=TdYanJ5ibkW0AVRunaz0rw@mail.gmail.com>
References: <CAP2=TMgQ7bF8N=6nhwk2i5=ZNNrpVR-+7fWN7Kmyj2mDHGUxPA@mail.gmail.com>
	<CAMYG4GnULoUjZYRoamfCUOkXsi6Bk-85o7BBy4W3kf4Fevq70w@mail.gmail.com>
	<CAP2=TMjqRDE1nBE_Z45+difnnBm=TdYanJ5ibkW0AVRunaz0rw@mail.gmail.com>
Message-ID: <CAMYG4GmdFTocO5JDH3orgQ3WrP0bje0VydE0PNt51hrLPeSspQ@mail.gmail.com>

On Wed, May 6, 2015 at 2:07 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Matt,
>
> Yes I would actually use a[] in this case. Back then, I think I was trying
> to take the derivative of the auxiliary which i had declared to be
> piece-wise constant.
>
> The auxiliary a[] is the gradient of pressure (times permeability over
> viscosity) obtained from solving the Darcy equation as a Laplacian. This
> part is solved separately, projected as cell-wise velocity, and inputted as
> an auxiliary for the advection-diffusion problem.
>
> Speaking of which, have you had a chance to look into
> DMPlexProjectFieldLocal(...) not projecting dirichlet BC constraints? I
> realize that the "small example" I gave you wasn't small at all, so I can
> send you a smaller and simpler one if needed.
>

For some reason, I thought that was taken care of. Please resend. Sorry
about that.

  Thanks,

     Matt


> Thanks,
> Justin
>
> On Tue, May 5, 2015 at 3:34 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Sat, Apr 4, 2015 at 11:54 AM, Justin Chang <jchang27 at uh.edu> wrote:
>>
>>> Hi everyone,
>>>
>>> I want to include advection into my diffusion FEM code (I am assuming a
>>> small Peclet number so stability isn't an issue ATM). That is I want to
>>> incorporate the second term as a pointwise function:
>>>
>>> du/dt + v * grad[c] - div[grad[c]] = f
>>>
>>> Where v is the velocity (obtained from the auxiliary term a_x[]). For
>>> the residual, would it be of the following form:
>>>
>>
>> 1) I would think you would use a[] instead. What is your velocity the
>> gradient of?
>>
>>
>>> f0_u(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar
>>> u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
>>> a_x[], const PetscReal x[], PetscScalar f0[]) {
>>>
>>>   PetscInt d;
>>>   f[0] = u_t[0];
>>>   for (d = 0; d < spatialDim; ++d) f0[0] += a_x[d]*u_x[d];
>>>
>>> }
>>>
>>> What about the jacobian? My guess would be to use g1(...) but what would
>>> the inside of this function be?
>>>
>>
>> Yes it would be g1. The indices for the output are f,g,dg. I am guessing
>> that c is a scalar, so f = {0}, g = {0}, dg = {0, 1}
>> for 2D, so g1 would have two terms,
>>
>>   g1[0] = a[0];
>>   g1[1] = a[1];
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>>
>>>
>>> --
>>> Justin Chang
>>> PhD Candidate, Civil Engineering - Computational Sciences
>>> University of Houston, Department of Civil and Environmental Engineering
>>> Houston, TX 77004
>>> (512) 963-3262
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/b877e320/attachment.html>

From knepley at gmail.com  Wed May  6 08:10:17 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 08:10:17 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87twvqgccy.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
Message-ID: <CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>

On Wed, May 6, 2015 at 6:43 AM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > Thank you guys for your responses. If I want to estimate the number of
> > bytes that come down, would -memory_info give me that information?
>
> No.  There is no substitute for a performance model.  Hardware
> performance counters can tell you something, but without a cache-aware
> performance model, it can give you information that is off by an order
> of magnitude or more.


I think the idea is to be explicit. I would just use something like

  # Vecs * Vec size * 8 bytes/double + <same for Mat>

and forget about small stuff. This model is uncached, and you just state
that up
front. Someone else can take the data and produce a more nuanced model.

You can use the Flop counts (not Flop/s) from PETSc,  and get a rough
estimate
of flop/byte ratio.

> And with this information plus the total number of logged flops, i can get
> > the ratio of flop to bytes and hence the (crude estimation of) upper
> bound
> > FLOPS/s based on the reported stream BW?
>
> Flop/s is logged (for PETSc operations) and you can use hardware
> performance counters for that (much better than for bandwidth, but still
> erroneous due to speculative execution and incomplete/gratuitous
> vectorization).  But to compute an arithmetic intensity or find
> performance bounds based on memory bandwidth considerations, there is no
> substitute for understanding what you are doing.  Sorry.
>

Do not let Jed's pessimism get you down.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/9f2003e3/attachment-0001.html>

From jed at jedbrown.org  Wed May  6 08:29:13 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 06 May 2015 09:29:13 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
Message-ID: <87oalxhm1i.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> I think the idea is to be explicit. I would just use something like
>
>   # Vecs * Vec size * 8 bytes/double + <same for Mat>
>
> and forget about small stuff. This model is uncached, 

This is for a perfect cache model -- each byte of the data structures
needs to be fetched from DRAM only once.

Make sure that "# Vecs" actually means the number that are accessed.
For example, GMRES accesses an increasing number of vectors on each
iteration (until a restart).

> and you just state that up front. Someone else can take the data and
> produce a more nuanced model.
>
> You can use the Flop counts (not Flop/s) from PETSc,  and get a rough
> estimate
> of flop/byte ratio.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/038a9036/attachment.pgp>

From knepley at gmail.com  Wed May  6 08:34:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 08:34:04 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87oalxhm1i.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
Message-ID: <CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>

On Wed, May 6, 2015 at 8:29 AM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > I think the idea is to be explicit. I would just use something like
> >
> >   # Vecs * Vec size * 8 bytes/double + <same for Mat>
> >
> > and forget about small stuff. This model is uncached,
>
> This is for a perfect cache model -- each byte of the data structures
> needs to be fetched from DRAM only once.
>

I meant uncached, in which you count # Vecs for any operation you are
doing. If you count # Vecs for the whole program, then you have perfect
cache.

  Matt


> Make sure that "# Vecs" actually means the number that are accessed.
> For example, GMRES accesses an increasing number of vectors on each
> iteration (until a restart).
>
> > and you just state that up front. Someone else can take the data and
> > produce a more nuanced model.
> >
> > You can use the Flop counts (not Flop/s) from PETSc,  and get a rough
> > estimate
> > of flop/byte ratio.
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/ed3ff1c1/attachment.html>

From jychang48 at gmail.com  Wed May  6 10:01:16 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 10:01:16 -0500
Subject: [petsc-users] advection-diffusion pointwise function for FEM
In-Reply-To: <CAMYG4GmdFTocO5JDH3orgQ3WrP0bje0VydE0PNt51hrLPeSspQ@mail.gmail.com>
References: <CAP2=TMgQ7bF8N=6nhwk2i5=ZNNrpVR-+7fWN7Kmyj2mDHGUxPA@mail.gmail.com>
	<CAMYG4GnULoUjZYRoamfCUOkXsi6Bk-85o7BBy4W3kf4Fevq70w@mail.gmail.com>
	<CAP2=TMjqRDE1nBE_Z45+difnnBm=TdYanJ5ibkW0AVRunaz0rw@mail.gmail.com>
	<CAMYG4GmdFTocO5JDH3orgQ3WrP0bje0VydE0PNt51hrLPeSspQ@mail.gmail.com>
Message-ID: <CAP2=TMj60JqA5p5zBWHWEWq5uh4ZeC=z9mkrNJLjYK9MV5Txrw@mail.gmail.com>

Never mind I think I figured it out. Will let you know if I think something
is wrong. Thanks anyway

On Wed, May 6, 2015 at 8:06 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 6, 2015 at 2:07 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Matt,
>>
>> Yes I would actually use a[] in this case. Back then, I think I was
>> trying to take the derivative of the auxiliary which i had declared to be
>> piece-wise constant.
>>
>> The auxiliary a[] is the gradient of pressure (times permeability over
>> viscosity) obtained from solving the Darcy equation as a Laplacian. This
>> part is solved separately, projected as cell-wise velocity, and inputted as
>> an auxiliary for the advection-diffusion problem.
>>
>> Speaking of which, have you had a chance to look into
>> DMPlexProjectFieldLocal(...) not projecting dirichlet BC constraints? I
>> realize that the "small example" I gave you wasn't small at all, so I can
>> send you a smaller and simpler one if needed.
>>
>
> For some reason, I thought that was taken care of. Please resend. Sorry
> about that.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Justin
>>
>> On Tue, May 5, 2015 at 3:34 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, Apr 4, 2015 at 11:54 AM, Justin Chang <jchang27 at uh.edu> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I want to include advection into my diffusion FEM code (I am assuming a
>>>> small Peclet number so stability isn't an issue ATM). That is I want to
>>>> incorporate the second term as a pointwise function:
>>>>
>>>> du/dt + v * grad[c] - div[grad[c]] = f
>>>>
>>>> Where v is the velocity (obtained from the auxiliary term a_x[]). For
>>>> the residual, would it be of the following form:
>>>>
>>>
>>> 1) I would think you would use a[] instead. What is your velocity the
>>> gradient of?
>>>
>>>
>>>> f0_u(const PetscScalar u[], const PetscScalar u_t[], const PetscScalar
>>>> u_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
>>>> a_x[], const PetscReal x[], PetscScalar f0[]) {
>>>>
>>>>   PetscInt d;
>>>>   f[0] = u_t[0];
>>>>   for (d = 0; d < spatialDim; ++d) f0[0] += a_x[d]*u_x[d];
>>>>
>>>> }
>>>>
>>>> What about the jacobian? My guess would be to use g1(...) but what
>>>> would the inside of this function be?
>>>>
>>>
>>> Yes it would be g1. The indices for the output are f,g,dg. I am guessing
>>> that c is a scalar, so f = {0}, g = {0}, dg = {0, 1}
>>> for 2D, so g1 would have two terms,
>>>
>>>   g1[0] = a[0];
>>>   g1[1] = a[1];
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Thanks,
>>>>
>>>>
>>>> --
>>>> Justin Chang
>>>> PhD Candidate, Civil Engineering - Computational Sciences
>>>> University of Houston, Department of Civil and Environmental Engineering
>>>> Houston, TX 77004
>>>> (512) 963-3262
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/2d75a652/attachment.html>

From jychang48 at gmail.com  Wed May  6 10:29:14 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 10:29:14 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
Message-ID: <CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>

Perhaps I should explain what my ultimate goal is.

I have two sets of code. First is the classical FEM which gives numerically
"incorrect answers" and the classical FEM employing convex optimization
through TAO which fixes these "incorrect answers". We have shown
numerically that the latter should always be the way to solve these types
of problems but now I want to compare the computational performance between
these two methodologies for large problems.

I already have speedup/strong scaling results that essentially depict the
difference between the KSPSolve() and TaoSolve(). However, I have been told
by someone that strong-scaling isn't enough - that I should somehow include
something to show the "efficiency" of these two methodologies. That is, how
much of the wall-clock time reported by these two very different solvers is
spent doing useful work.

Is such an "efficiency" metric necessary to report in addition to
strong-scaling results? The overall computational framework is the same for
both problems, the only difference being one uses a linear solver and the
other uses an optimization solver. My first thought was to use PAPI to
include hardware counters, but these are notoriously inaccurate. Then I
thought about simply reporting the manual FLOPS and FLOPS/s via PETSc, but
these metrics ignore memory bandwidth. And so here I am looking at the idea
of implementing the Roofline model, but now I am wondering if any of this
is worth the trouble.

Thanks,
Justin


On Wed, May 6, 2015 at 8:34 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 6, 2015 at 8:29 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>> > I think the idea is to be explicit. I would just use something like
>> >
>> >   # Vecs * Vec size * 8 bytes/double + <same for Mat>
>> >
>> > and forget about small stuff. This model is uncached,
>>
>> This is for a perfect cache model -- each byte of the data structures
>> needs to be fetched from DRAM only once.
>>
>
> I meant uncached, in which you count # Vecs for any operation you are
> doing. If you count # Vecs for the whole program, then you have perfect
> cache.
>
>   Matt
>
>
>> Make sure that "# Vecs" actually means the number that are accessed.
>> For example, GMRES accesses an increasing number of vectors on each
>> iteration (until a restart).
>>
>> > and you just state that up front. Someone else can take the data and
>> > produce a more nuanced model.
>> >
>> > You can use the Flop counts (not Flop/s) from PETSc,  and get a rough
>> > estimate
>> > of flop/byte ratio.
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/b9edfabe/attachment.html>

From knepley at gmail.com  Wed May  6 10:39:30 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 10:39:30 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
Message-ID: <CAMYG4Gm8ij9wKz=kiogRz7w3noXv=-LAa_aSW8ohdc+WKS6QKQ@mail.gmail.com>

On Wed, May 6, 2015 at 10:29 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Perhaps I should explain what my ultimate goal is.
>
> I have two sets of code. First is the classical FEM which gives
> numerically "incorrect answers" and the classical FEM employing convex
> optimization through TAO which fixes these "incorrect answers". We have
> shown numerically that the latter should always be the way to solve these
> types of problems but now I want to compare the computational performance
> between these two methodologies for large problems.
>
> I already have speedup/strong scaling results that essentially depict the
> difference between the KSPSolve() and TaoSolve(). However, I have been told
> by someone that strong-scaling isn't enough - that I should somehow include
> something to show the "efficiency" of these two methodologies. That is, how
> much of the wall-clock time reported by these two very different solvers is
> spent doing useful work.
>
> Is such an "efficiency" metric necessary to report in addition to
> strong-scaling results? The overall computational framework is the same for
> both problems, the only difference being one uses a linear solver and the
> other uses an optimization solver. My first thought was to use PAPI to
> include hardware counters, but these are notoriously inaccurate. Then I
> thought about simply reporting the manual FLOPS and FLOPS/s via PETSc, but
> these metrics ignore memory bandwidth. And so here I am looking at the idea
> of implementing the Roofline model, but now I am wondering if any of this
> is worth the trouble.
>

I think its always best to start with a goal, and then explain how to
achieve that goal.

If your goal is "Run a particular problem as fast as possible", then strong
scaling is a good metric. It shows
you how much time improvement you get by adding more processors. However,
suppose I just put a TON
of useless work into my solver which is perfectly parallel, like factoring
a bunch of large numbers. Now I can
show perfect strong scaling, but this is not the most _efficient_ way to
implement this solver.

Now there are at least two kinds of efficiency, hardware efficiency and
algorithmic efficiency. For both you have
to have some conceptual model of the computation. Then I can say "This is
how long I expect things to take in
a perfect world", and "this is how long your algorithm takes". Also, I can
say "This is how fast it should run on
this hardware in a perfect world", and "this is how fast it runs on my
machine".

Both measures are interesting and useful. As Jed points out, you must have
some kind of model for the computation
in order to make claims about "efficiency". For example, I expect VecDot()
for length N vectors to use 2N flops and
pull 2N data, and do one reduction. You can imagine algorithms which do
this slightly differently. You can imagine
hardware where this is flop limited and where this is memory bandwidth
limited. I can talk about these tradeoffs once
I have the simple computational model.

Does this help?

  Thanks,

    Matt


> Thanks,
> Justin
>
>
> On Wed, May 6, 2015 at 8:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, May 6, 2015 at 8:29 AM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Matthew Knepley <knepley at gmail.com> writes:
>>> > I think the idea is to be explicit. I would just use something like
>>> >
>>> >   # Vecs * Vec size * 8 bytes/double + <same for Mat>
>>> >
>>> > and forget about small stuff. This model is uncached,
>>>
>>> This is for a perfect cache model -- each byte of the data structures
>>> needs to be fetched from DRAM only once.
>>>
>>
>> I meant uncached, in which you count # Vecs for any operation you are
>> doing. If you count # Vecs for the whole program, then you have perfect
>> cache.
>>
>>   Matt
>>
>>
>>> Make sure that "# Vecs" actually means the number that are accessed.
>>> For example, GMRES accesses an increasing number of vectors on each
>>> iteration (until a restart).
>>>
>>> > and you just state that up front. Someone else can take the data and
>>> > produce a more nuanced model.
>>> >
>>> > You can use the Flop counts (not Flop/s) from PETSc,  and get a rough
>>> > estimate
>>> > of flop/byte ratio.
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/f444f7d4/attachment-0001.html>

From jed at jedbrown.org  Wed May  6 10:42:22 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 06 May 2015 11:42:22 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
Message-ID: <87lhh1hfvl.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
>> This is for a perfect cache model -- each byte of the data structures
>> needs to be fetched from DRAM only once.
>>
>
> I meant uncached, in which you count # Vecs for any operation you are
> doing. 

Wrong, you're describing a perfect cache model [1].  The initial data
does not reside in cache, but you only need to fetch each byte from DRAM
once.  The reality for larger problem sizes is that the entire wavefront
is not resident (e.g., perhaps because matrix entries evict vector
entries) and thus a single SpMV needs to re-load some vector entries.
For heavier matrices, this does not significantly change the bandwidth
requirements.  If the access pattern is predictable (i.e., you often get
lucky if you choose a locality-preserving order) then the perfect-cache
pure bandwidth model is good.

> If you count # Vecs for the whole program, then you have perfect
> cache.

[1] Or we have different definitions of "perfect cache", but I don't
think it's useful to discuss DRAM bandwidth if all data is resident in
cache, so I'm referring to perfect caching of inputs that are not
resident in cache.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/4ac84bc6/attachment.pgp>

From knepley at gmail.com  Wed May  6 11:26:20 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 11:26:20 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87lhh1hfvl.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<87lhh1hfvl.fsf@jedbrown.org>
Message-ID: <CAMYG4Gm9_L5B5yH-mHEPR0bi=iwsvHw9JTmTc2ew_6YnT0PutA@mail.gmail.com>

On Wed, May 6, 2015 at 10:42 AM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> >> This is for a perfect cache model -- each byte of the data structures
> >> needs to be fetched from DRAM only once.
> >>
> >
> > I meant uncached, in which you count # Vecs for any operation you are
> > doing.
>
> Wrong, you're describing a perfect cache model [1].  The initial data
> does not reside in cache, but you only need to fetch each byte from DRAM
> once.  The reality for larger problem sizes is that the entire wavefront
> is not resident (e.g., perhaps because matrix entries evict vector
> entries) and thus a single SpMV needs to re-load some vector entries.
> For heavier matrices, this does not significantly change the bandwidth
> requirements.  If the access pattern is predictable (i.e., you often get
> lucky if you choose a locality-preserving order) then the perfect-cache
> pure bandwidth model is good.
>
> > If you count # Vecs for the whole program, then you have perfect
> > cache.
>
> [1] Or we have different definitions of "perfect cache", but I don't
> think it's useful to discuss DRAM bandwidth if all data is resident in
> cache, so I'm referring to perfect caching of inputs that are not
> resident in cache.
>

Yes, this is a language problem only.

  Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/8bde59c1/attachment.html>

From jed at jedbrown.org  Wed May  6 11:28:25 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 06 May 2015 12:28:25 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
Message-ID: <87d22dhdqu.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:
> I already have speedup/strong scaling results that essentially depict the
> difference between the KSPSolve() and TaoSolve(). However, I have been told
> by someone that strong-scaling isn't enough - that I should somehow include
> something to show the "efficiency" of these two methodologies. 

"Efficiency" is irrelevant if one is wrong.  Can you set up a problem
where both get the right answer and vary a parameter to get to the case
where one fails?  Then you can look at efficiency for a given accuracy
(and you might have to refine the grid differently) as you vary the
parameter.

It's really hard to demonstrate that an implicit solver is optimal in
terms of mathematical convergence rate.  Improvements there can dwarf
any differences in implementation efficiency.

> That is, how much of the wall-clock time reported by these two very
> different solvers is spent doing useful work.
>
> Is such an "efficiency" metric necessary to report in addition to
> strong-scaling results? The overall computational framework is the same for
> both problems, the only difference being one uses a linear solver and the
> other uses an optimization solver. My first thought was to use PAPI to
> include hardware counters, but these are notoriously inaccurate. Then I
> thought about simply reporting the manual FLOPS and FLOPS/s via PETSc, but
> these metrics ignore memory bandwidth. And so here I am looking at the idea
> of implementing the Roofline model, but now I am wondering if any of this
> is worth the trouble.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/10b2d5de/attachment.pgp>

From jychang48 at gmail.com  Wed May  6 11:41:27 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 11:41:27 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87d22dhdqu.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
Message-ID: <CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>

I suppose I have two objectives that I think are achievable within PETSc
means:

1) How much wall-clock time can be reduced as you increase the number of
processors. I have strong-scaling and parallel efficiency metrics that
convey this.

2) The "optimal" problem size for these two methods/solvers are. What I
mean by this is, at what point do I achieve the maximum FLOPS/s. If
starting off with a really small problem then this metric should increase
with problem size. My hypothesis is that as problem size increases, the
ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
accessing main memory, etc) to performing work also increases, and the
reported FLOPS/s should start decreasing at some point. "Efficiency" in
this context simply means the highest possible FLOPS/s.

Does that make sense and/or is "interesting" enough?

Thanks,
Justin

On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
> > I already have speedup/strong scaling results that essentially depict the
> > difference between the KSPSolve() and TaoSolve(). However, I have been
> told
> > by someone that strong-scaling isn't enough - that I should somehow
> include
> > something to show the "efficiency" of these two methodologies.
>
> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
> where both get the right answer and vary a parameter to get to the case
> where one fails?  Then you can look at efficiency for a given accuracy
> (and you might have to refine the grid differently) as you vary the
> parameter.
>
> It's really hard to demonstrate that an implicit solver is optimal in
> terms of mathematical convergence rate.  Improvements there can dwarf
> any differences in implementation efficiency.
>
> > That is, how much of the wall-clock time reported by these two very
> > different solvers is spent doing useful work.
> >
> > Is such an "efficiency" metric necessary to report in addition to
> > strong-scaling results? The overall computational framework is the same
> for
> > both problems, the only difference being one uses a linear solver and the
> > other uses an optimization solver. My first thought was to use PAPI to
> > include hardware counters, but these are notoriously inaccurate. Then I
> > thought about simply reporting the manual FLOPS and FLOPS/s via PETSc,
> but
> > these metrics ignore memory bandwidth. And so here I am looking at the
> idea
> > of implementing the Roofline model, but now I am wondering if any of this
> > is worth the trouble.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/1fd9e99d/attachment.html>

From knepley at gmail.com  Wed May  6 11:48:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 11:48:03 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
Message-ID: <CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>

On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com> wrote:

> I suppose I have two objectives that I think are achievable within PETSc
> means:
>
> 1) How much wall-clock time can be reduced as you increase the number of
> processors. I have strong-scaling and parallel efficiency metrics that
> convey this.
>
> 2) The "optimal" problem size for these two methods/solvers are. What I
> mean by this is, at what point do I achieve the maximum FLOPS/s. If
> starting off with a really small problem then this metric should increase
> with problem size. My hypothesis is that as problem size increases, the
> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
> accessing main memory, etc) to performing work also increases, and the
> reported FLOPS/s should start decreasing at some point. "Efficiency" in
> this context simply means the highest possible FLOPS/s.
>
> Does that make sense and/or is "interesting" enough?
>

I think 2) is not really that interesting because

  a) it is so easily gamed. Just stick in high flop count operations, like
DGEMM.

  b) Time really matters to people who run the code, but flops never do.

  c) Floating point performance is not your limiting factor for time

I think it would be much more interesting, and no more work to

  a) Model the flop/byte \beta ratio simply

  b) Report how close you get to the max performance given \beta on your
machine

  Thanks,

     Matt


> Thanks,
> Justin
>
> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Justin Chang <jychang48 at gmail.com> writes:
>> > I already have speedup/strong scaling results that essentially depict
>> the
>> > difference between the KSPSolve() and TaoSolve(). However, I have been
>> told
>> > by someone that strong-scaling isn't enough - that I should somehow
>> include
>> > something to show the "efficiency" of these two methodologies.
>>
>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>> where both get the right answer and vary a parameter to get to the case
>> where one fails?  Then you can look at efficiency for a given accuracy
>> (and you might have to refine the grid differently) as you vary the
>> parameter.
>>
>> It's really hard to demonstrate that an implicit solver is optimal in
>> terms of mathematical convergence rate.  Improvements there can dwarf
>> any differences in implementation efficiency.
>>
>> > That is, how much of the wall-clock time reported by these two very
>> > different solvers is spent doing useful work.
>> >
>> > Is such an "efficiency" metric necessary to report in addition to
>> > strong-scaling results? The overall computational framework is the same
>> for
>> > both problems, the only difference being one uses a linear solver and
>> the
>> > other uses an optimization solver. My first thought was to use PAPI to
>> > include hardware counters, but these are notoriously inaccurate. Then I
>> > thought about simply reporting the manual FLOPS and FLOPS/s via PETSc,
>> but
>> > these metrics ignore memory bandwidth. And so here I am looking at the
>> idea
>> > of implementing the Roofline model, but now I am wondering if any of
>> this
>> > is worth the trouble.
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/017510b9/attachment-0001.html>

From E.Mueller at bath.ac.uk  Wed May  6 12:08:33 2015
From: E.Mueller at bath.ac.uk (Eike Mueller)
Date: Wed, 6 May 2015 17:08:33 +0000
Subject: [petsc-users] Seting advanced smoothers for BoomerAMG through
 PETSc
In-Reply-To: <14B9A4F8-5114-4579-BF5B-BBDFE265B1C8@mcs.anl.gov>
References: <1E962B7A-BE00-43E3-A202-37E64BD58371@bath.ac.uk>
	<14B9A4F8-5114-4579-BF5B-BBDFE265B1C8@mcs.anl.gov>
Message-ID: <1AB3CCAC-E610-4236-BC8F-6487CD95617D@bath.ac.uk>

Dear Barry,

thanks, I managed to get that working for HYPRE_BoomerAMGSetSmoothType(), but I need to set more options to control the different smoothers in detail. I will try to get the changes into PETSc once I?m done.

Eike

> On 6 May 2015, at 13:27, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>  Eike,
> 
>   You'll need to edit src/ksp/pc/impls/hypre/hypre.c and add an option to set this (where you see all the other BoomerAMG options). Then run make in
> the root PETSc directory.
> 
>   If you do this in the development copy of PETSc and make a pull request (see https://bitbucket.org/petsc/petsc/wiki/Home Contributing to PETSc) then all future users can benefit from your additions.
> 
>  Barry
> 
>> On May 6, 2015, at 3:14 AM, Eike Mueller <E.Mueller at bath.ac.uk> wrote:
>> 
>> Dear PETSc,
>> 
>> I?m trying to use hypre?s BoomerAMG preconditioner through PETSc. However, I would like to use a non-standard smoother, namely an incomplete LU factorisation, which is implemented in hypre?s Euclid (that?s because the problem is highly anisotropic, and GAMG worked very weill with ILU smoother). In native hypre I can set this with HYPRE_BoomerAMGSetSmoothType(). However, I can?t see how I can pass this through the PETSc interface, since the only boomeramg options I can set are listed below. As far as I can see they allow me to change the pointwise smoother via HYPRE_BoomerAMGSetRelaxType().
>> 
>> Does anyone have any advice on what I can do to use Euclid as a smoother via setting the correct PETSc options?
>> 
>> Thank you very much,
>> 
>> Eike
>> 
>> eikemueller at 138-38-173-198 $ ./ex6 -f -f helmholtz-sphere.dat  -ksp_type cg -ksp_convergence_test skip -ksp_max_it 2  -ksp_monitor -table -pc_hypre_type boomeramg -pc_type hypre -log_summary -ksp_view -help | grep boomeramg
>>  -pc_hypre_type <boomeramg> (choose one of) pilut parasails boomeramg ams (PCHYPRESetType)
>>  -pc_hypre_boomeramg_cycle_type <V> (choose one of) V W (None)
>>  -pc_hypre_boomeramg_max_levels <25>: Number of levels (of grids) allowed (None)
>>  -pc_hypre_boomeramg_max_iter <1>: Maximum iterations used PER hypre call (None)
>>  -pc_hypre_boomeramg_tol <0>: Convergence tolerance PER hypre call (0.0 = use a fixed number of iterations) (None)
>>  -pc_hypre_boomeramg_truncfactor <0>: Truncation factor for interpolation (0=no truncation) (None)
>>  -pc_hypre_boomeramg_P_max <0>: Max elements per row for interpolation operator (0=unlimited) (None)
>>  -pc_hypre_boomeramg_agg_nl <0>: Number of levels of aggressive coarsening (None)
>>  -pc_hypre_boomeramg_agg_num_paths <1>: Number of paths for aggressive coarsening (None)
>>  -pc_hypre_boomeramg_strong_threshold <0.25>: Threshold for being strongly connected (None)
>>  -pc_hypre_boomeramg_max_row_sum <0.9>: Maximum row sum (None)
>>  -pc_hypre_boomeramg_grid_sweeps_all <1>: Number of sweeps for the up and down grid levels (None)
>>  -pc_hypre_boomeramg_grid_sweeps_down <1>: Number of sweeps for the down cycles (None)
>>  -pc_hypre_boomeramg_grid_sweeps_up <1>: Number of sweeps for the up cycles (None)
>>  -pc_hypre_boomeramg_grid_sweeps_coarse <1>: Number of sweeps for the coarse level (None)
>>  -pc_hypre_boomeramg_relax_type_all <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>>  -pc_hypre_boomeramg_relax_type_down <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>>  -pc_hypre_boomeramg_relax_type_up <symmetric-SOR/Jacobi> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>>  -pc_hypre_boomeramg_relax_type_coarse <Gaussian-elimination> (choose one of) Jacobi sequential-Gauss-Seidel seqboundary-Gauss-Seidel SOR/Jacobi backward-SOR/Jacobi  symmetric-SOR/Jacobi  l1scaled-SOR/Jacobi Gaussian-elimination      CG Chebyshev FCF-Jacobi l1scaled-Jacobi (None)
>>  -pc_hypre_boomeramg_relax_weight_all <1>: Relaxation weight for all levels (0 = hypre estimates, -k = determined with k CG steps) (None)
>>  -pc_hypre_boomeramg_relax_weight_level <1>: Set the relaxation weight for a particular level (weight,level) (None)
>>  -pc_hypre_boomeramg_outer_relax_weight_all <1>: Outer relaxation weight for all levels (-k = determined with k CG steps) (None)
>>  -pc_hypre_boomeramg_outer_relax_weight_level <1>: Set the outer relaxation weight for a particular level (weight,level) (None)
>>  -pc_hypre_boomeramg_no_CF: <FALSE> Do not use CF-relaxation (None)
>>  -pc_hypre_boomeramg_measure_type <local> (choose one of) local global (None)
>>  -pc_hypre_boomeramg_coarsen_type <Falgout> (choose one of) CLJP Ruge-Stueben  modifiedRuge-Stueben   Falgout  PMIS  HMIS (None)
>>  -pc_hypre_boomeramg_interp_type <classical> (choose one of) classical   direct multipass multipass-wts ext+i ext+i-cc standard standard-wts   FF FF1 (None)
>>  -pc_hypre_boomeramg_print_statistics: Print statistics (None)
>>  -pc_hypre_boomeramg_print_debug: Print debug information (None)
>>  -pc_hypre_boomeramg_nodal_coarsen: <FALSE> HYPRE_BoomerAMGSetNodal() (None)
>>  -pc_hypre_boomeramg_nodal_relaxation: <FALSE> Nodal relaxation via Schwarz (None)
>> -pc_hypre_type boomeramg
>> 
>> --
>> 
>> Dr Eike Hermann Mueller
>> Lecturer in Scientific Computing
>> 
>> Department of Mathematical Sciences
>> University of Bath
>> Bath BA2 7AY, United Kingdom
>> 
>> +44 1225 38 6241
>> e.mueller at bath.ac.uk
>> http://people.bath.ac.uk/em459/
>> 
>> 
>> 
>> 
>> 
> 


From jychang48 at gmail.com  Wed May  6 13:28:03 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 13:28:03 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
Message-ID: <CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>

Jed,

I am working with anisotropic diffusion and most standard numerical
formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
on this. What we have seen people do is simply "ignore" or chop off these
values but to us that is a complete and utter abomination. My goal here is
to show that our proposed methodologies work by leveraging on the
capabilities within PETSc and TAO and to also show how computationally
expensive it is compared to solving the same problem using the standard
Galerkin method.

Matt,

Okay, so then I guess I still have questions regarding how to obtain the
bytes. How exactly would I count all the number of Vecs and their
respective sizes, because it seems all the DMPlex related functions create
many vectors. Or do I only count the DM created vectors used for my
solution vector, residual, lower/upper bound, optimization routines, etc?

And when you say "forget about small stuff", does that include all the
DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
all the jazz that goes on within the FE/discretization routines?

Lastly, for a Matrix, wouldn't I just get the number of bytes from the
memory usage section in -log_summary?

Thanks,
Justin

On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> I suppose I have two objectives that I think are achievable within PETSc
>> means:
>>
>> 1) How much wall-clock time can be reduced as you increase the number of
>> processors. I have strong-scaling and parallel efficiency metrics that
>> convey this.
>>
>> 2) The "optimal" problem size for these two methods/solvers are. What I
>> mean by this is, at what point do I achieve the maximum FLOPS/s. If
>> starting off with a really small problem then this metric should increase
>> with problem size. My hypothesis is that as problem size increases, the
>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>> accessing main memory, etc) to performing work also increases, and the
>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>> this context simply means the highest possible FLOPS/s.
>>
>> Does that make sense and/or is "interesting" enough?
>>
>
> I think 2) is not really that interesting because
>
>   a) it is so easily gamed. Just stick in high flop count operations, like
> DGEMM.
>
>   b) Time really matters to people who run the code, but flops never do.
>
>   c) Floating point performance is not your limiting factor for time
>
> I think it would be much more interesting, and no more work to
>
>   a) Model the flop/byte \beta ratio simply
>
>   b) Report how close you get to the max performance given \beta on your
> machine
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Justin
>>
>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Justin Chang <jychang48 at gmail.com> writes:
>>> > I already have speedup/strong scaling results that essentially depict
>>> the
>>> > difference between the KSPSolve() and TaoSolve(). However, I have been
>>> told
>>> > by someone that strong-scaling isn't enough - that I should somehow
>>> include
>>> > something to show the "efficiency" of these two methodologies.
>>>
>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>>> where both get the right answer and vary a parameter to get to the case
>>> where one fails?  Then you can look at efficiency for a given accuracy
>>> (and you might have to refine the grid differently) as you vary the
>>> parameter.
>>>
>>> It's really hard to demonstrate that an implicit solver is optimal in
>>> terms of mathematical convergence rate.  Improvements there can dwarf
>>> any differences in implementation efficiency.
>>>
>>> > That is, how much of the wall-clock time reported by these two very
>>> > different solvers is spent doing useful work.
>>> >
>>> > Is such an "efficiency" metric necessary to report in addition to
>>> > strong-scaling results? The overall computational framework is the
>>> same for
>>> > both problems, the only difference being one uses a linear solver and
>>> the
>>> > other uses an optimization solver. My first thought was to use PAPI to
>>> > include hardware counters, but these are notoriously inaccurate. Then I
>>> > thought about simply reporting the manual FLOPS and FLOPS/s via PETSc,
>>> but
>>> > these metrics ignore memory bandwidth. And so here I am looking at the
>>> idea
>>> > of implementing the Roofline model, but now I am wondering if any of
>>> this
>>> > is worth the trouble.
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/eecd5288/attachment.html>

From knepley at gmail.com  Wed May  6 13:38:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 13:38:09 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
Message-ID: <CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>

On Wed, May 6, 2015 at 1:28 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Jed,
>
> I am working with anisotropic diffusion and most standard numerical
> formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
> discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
> on this. What we have seen people do is simply "ignore" or chop off these
> values but to us that is a complete and utter abomination. My goal here is
> to show that our proposed methodologies work by leveraging on the
> capabilities within PETSc and TAO and to also show how computationally
> expensive it is compared to solving the same problem using the standard
> Galerkin method.
>
> Matt,
>
> Okay, so then I guess I still have questions regarding how to obtain the
> bytes. How exactly would I count all the number of Vecs and their
> respective sizes, because it seems all the DMPlex related functions create
> many vectors. Or do I only count the DM created vectors used for my
> solution vector, residual, lower/upper bound, optimization routines, etc?
>

This is laborious. You would build up from the small stuff. So a Krylov
solver have MatMult, for which there is an analysis in the paper with
Dinesh/Bill/Barry/David, and
Vec ops which are easy. This is a lot of counting, especially if you have a
TAO solver in there. I would make sure you really care.


> And when you say "forget about small stuff", does that include all the
> DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
> all the jazz that goes on within the FE/discretization routines?
>

Yep.


> Lastly, for a Matrix, wouldn't I just get the number of bytes from the
> memory usage section in -log_summary?
>

That is a good way. You can also ask MatInfo how many nonzeros the matrix
has.

   Matt


> Thanks,
> Justin
>
> On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com>
>> wrote:
>>
>>> I suppose I have two objectives that I think are achievable within PETSc
>>> means:
>>>
>>> 1) How much wall-clock time can be reduced as you increase the number of
>>> processors. I have strong-scaling and parallel efficiency metrics that
>>> convey this.
>>>
>>> 2) The "optimal" problem size for these two methods/solvers are. What I
>>> mean by this is, at what point do I achieve the maximum FLOPS/s. If
>>> starting off with a really small problem then this metric should increase
>>> with problem size. My hypothesis is that as problem size increases, the
>>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>>> accessing main memory, etc) to performing work also increases, and the
>>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>>> this context simply means the highest possible FLOPS/s.
>>>
>>> Does that make sense and/or is "interesting" enough?
>>>
>>
>> I think 2) is not really that interesting because
>>
>>   a) it is so easily gamed. Just stick in high flop count operations,
>> like DGEMM.
>>
>>   b) Time really matters to people who run the code, but flops never do.
>>
>>   c) Floating point performance is not your limiting factor for time
>>
>> I think it would be much more interesting, and no more work to
>>
>>   a) Model the flop/byte \beta ratio simply
>>
>>   b) Report how close you get to the max performance given \beta on your
>> machine
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Justin Chang <jychang48 at gmail.com> writes:
>>>> > I already have speedup/strong scaling results that essentially depict
>>>> the
>>>> > difference between the KSPSolve() and TaoSolve(). However, I have
>>>> been told
>>>> > by someone that strong-scaling isn't enough - that I should somehow
>>>> include
>>>> > something to show the "efficiency" of these two methodologies.
>>>>
>>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>>>> where both get the right answer and vary a parameter to get to the case
>>>> where one fails?  Then you can look at efficiency for a given accuracy
>>>> (and you might have to refine the grid differently) as you vary the
>>>> parameter.
>>>>
>>>> It's really hard to demonstrate that an implicit solver is optimal in
>>>> terms of mathematical convergence rate.  Improvements there can dwarf
>>>> any differences in implementation efficiency.
>>>>
>>>> > That is, how much of the wall-clock time reported by these two very
>>>> > different solvers is spent doing useful work.
>>>> >
>>>> > Is such an "efficiency" metric necessary to report in addition to
>>>> > strong-scaling results? The overall computational framework is the
>>>> same for
>>>> > both problems, the only difference being one uses a linear solver and
>>>> the
>>>> > other uses an optimization solver. My first thought was to use PAPI to
>>>> > include hardware counters, but these are notoriously inaccurate. Then
>>>> I
>>>> > thought about simply reporting the manual FLOPS and FLOPS/s via
>>>> PETSc, but
>>>> > these metrics ignore memory bandwidth. And so here I am looking at
>>>> the idea
>>>> > of implementing the Roofline model, but now I am wondering if any of
>>>> this
>>>> > is worth the trouble.
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/87eca075/attachment-0001.html>

From jychang48 at gmail.com  Wed May  6 14:28:47 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 6 May 2015 14:28:47 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
	<CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>
Message-ID: <CAP2=TMhBcTriQVCGtKmW=cj=hMhnBgrYthKMbAks6R1xNUz99Q@mail.gmail.com>

So basically I just need these types of operations:

VecTDot               60 1.0 1.6928e-05 1.0 2.69e+04 1.0 0.0e+00 0.0e+00
0.0e+00  0 14  0  0  0   0 14  0  0  0  1591
VecNorm               31 1.0 9.0599e-06 1.0 1.39e+04 1.0 0.0e+00 0.0e+00
0.0e+00  0  7  0  0  0   0  7  0  0  0  1536
VecCopy                3 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                36 1.0 9.4175e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecAXPY               60 1.0 1.7166e-05 1.0 2.70e+04 1.0 0.0e+00 0.0e+00
0.0e+00  0 14  0  0  0   0 14  0  0  0  1573
VecAYPX               29 1.0 1.9312e-05 1.0 1.30e+04 1.0 0.0e+00 0.0e+00
0.0e+00  0  7  0  0  0   0  7  0  0  0   676
VecWAXPY               1 1.0 1.9073e-06 1.0 2.25e+02 1.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0   118
VecPointwiseMult      31 1.0 1.8358e-05 1.0 6.98e+03 1.0 0.0e+00 0.0e+00
0.0e+00  0  4  0  0  0   0  4  0  0  0   380
MatMult               30 1.0 7.5340e-05 1.0 8.07e+04 1.0 0.0e+00 0.0e+00
0.0e+00  0 42  0  0  0   0 42  0  0  0  1071

Given the matrix size and the number of nonzeros, vector size of my
solution, and the number of calls to each of these vec ops, I can estimate
the total bytes transferred (loads/stores)?

For multiple processors, do I calculate the local or global sizes? If I
calculate the local sizes, then do I need to do an MPI_Allreduce (with
mpi_sum) of the TBT across all processors just like with total flops?

Do these vec/mat ops account for Dirichlet constraints? That is, should the
global/local size include those constraints?

Also, is there a way to extract the event count for these operations
besides dumping -log_summary each time?

Thanks,
Justin

On Wed, May 6, 2015 at 1:38 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 6, 2015 at 1:28 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Jed,
>>
>> I am working with anisotropic diffusion and most standard numerical
>> formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
>> discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
>> on this. What we have seen people do is simply "ignore" or chop off these
>> values but to us that is a complete and utter abomination. My goal here is
>> to show that our proposed methodologies work by leveraging on the
>> capabilities within PETSc and TAO and to also show how computationally
>> expensive it is compared to solving the same problem using the standard
>> Galerkin method.
>>
>> Matt,
>>
>> Okay, so then I guess I still have questions regarding how to obtain the
>> bytes. How exactly would I count all the number of Vecs and their
>> respective sizes, because it seems all the DMPlex related functions create
>> many vectors. Or do I only count the DM created vectors used for my
>> solution vector, residual, lower/upper bound, optimization routines, etc?
>>
>
> This is laborious. You would build up from the small stuff. So a Krylov
> solver have MatMult, for which there is an analysis in the paper with
> Dinesh/Bill/Barry/David, and
> Vec ops which are easy. This is a lot of counting, especially if you have
> a TAO solver in there. I would make sure you really care.
>
>
>> And when you say "forget about small stuff", does that include all the
>> DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
>> all the jazz that goes on within the FE/discretization routines?
>>
>
> Yep.
>
>
>> Lastly, for a Matrix, wouldn't I just get the number of bytes from the
>> memory usage section in -log_summary?
>>
>
> That is a good way. You can also ask MatInfo how many nonzeros the matrix
> has.
>
>    Matt
>
>
>> Thanks,
>> Justin
>>
>> On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> I suppose I have two objectives that I think are achievable within
>>>> PETSc means:
>>>>
>>>> 1) How much wall-clock time can be reduced as you increase the number
>>>> of processors. I have strong-scaling and parallel efficiency metrics that
>>>> convey this.
>>>>
>>>> 2) The "optimal" problem size for these two methods/solvers are. What I
>>>> mean by this is, at what point do I achieve the maximum FLOPS/s. If
>>>> starting off with a really small problem then this metric should increase
>>>> with problem size. My hypothesis is that as problem size increases, the
>>>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>>>> accessing main memory, etc) to performing work also increases, and the
>>>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>>>> this context simply means the highest possible FLOPS/s.
>>>>
>>>> Does that make sense and/or is "interesting" enough?
>>>>
>>>
>>> I think 2) is not really that interesting because
>>>
>>>   a) it is so easily gamed. Just stick in high flop count operations,
>>> like DGEMM.
>>>
>>>   b) Time really matters to people who run the code, but flops never do.
>>>
>>>   c) Floating point performance is not your limiting factor for time
>>>
>>> I think it would be much more interesting, and no more work to
>>>
>>>   a) Model the flop/byte \beta ratio simply
>>>
>>>   b) Report how close you get to the max performance given \beta on your
>>> machine
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Justin Chang <jychang48 at gmail.com> writes:
>>>>> > I already have speedup/strong scaling results that essentially
>>>>> depict the
>>>>> > difference between the KSPSolve() and TaoSolve(). However, I have
>>>>> been told
>>>>> > by someone that strong-scaling isn't enough - that I should somehow
>>>>> include
>>>>> > something to show the "efficiency" of these two methodologies.
>>>>>
>>>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>>>>> where both get the right answer and vary a parameter to get to the case
>>>>> where one fails?  Then you can look at efficiency for a given accuracy
>>>>> (and you might have to refine the grid differently) as you vary the
>>>>> parameter.
>>>>>
>>>>> It's really hard to demonstrate that an implicit solver is optimal in
>>>>> terms of mathematical convergence rate.  Improvements there can dwarf
>>>>> any differences in implementation efficiency.
>>>>>
>>>>> > That is, how much of the wall-clock time reported by these two very
>>>>> > different solvers is spent doing useful work.
>>>>> >
>>>>> > Is such an "efficiency" metric necessary to report in addition to
>>>>> > strong-scaling results? The overall computational framework is the
>>>>> same for
>>>>> > both problems, the only difference being one uses a linear solver
>>>>> and the
>>>>> > other uses an optimization solver. My first thought was to use PAPI
>>>>> to
>>>>> > include hardware counters, but these are notoriously inaccurate.
>>>>> Then I
>>>>> > thought about simply reporting the manual FLOPS and FLOPS/s via
>>>>> PETSc, but
>>>>> > these metrics ignore memory bandwidth. And so here I am looking at
>>>>> the idea
>>>>> > of implementing the Roofline model, but now I am wondering if any of
>>>>> this
>>>>> > is worth the trouble.
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/5ad261e5/attachment.html>

From knepley at gmail.com  Wed May  6 14:34:13 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 6 May 2015 14:34:13 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMhBcTriQVCGtKmW=cj=hMhnBgrYthKMbAks6R1xNUz99Q@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
	<CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>
	<CAP2=TMhBcTriQVCGtKmW=cj=hMhnBgrYthKMbAks6R1xNUz99Q@mail.gmail.com>
Message-ID: <CAMYG4GmgHe_XfqV0usFXgKCgfoDGLU-QE0+cqiD8hj0jg8b=7A@mail.gmail.com>

On Wed, May 6, 2015 at 2:28 PM, Justin Chang <jychang48 at gmail.com> wrote:

> So basically I just need these types of operations:
>
> VecTDot               60 1.0 1.6928e-05 1.0 2.69e+04 1.0 0.0e+00 0.0e+00
> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1591
> VecNorm               31 1.0 9.0599e-06 1.0 1.39e+04 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  7  0  0  0   0  7  0  0  0  1536
> VecCopy                3 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet                36 1.0 9.4175e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> VecAXPY               60 1.0 1.7166e-05 1.0 2.70e+04 1.0 0.0e+00 0.0e+00
> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1573
> VecAYPX               29 1.0 1.9312e-05 1.0 1.30e+04 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  7  0  0  0   0  7  0  0  0   676
> VecWAXPY               1 1.0 1.9073e-06 1.0 2.25e+02 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0   118
> VecPointwiseMult      31 1.0 1.8358e-05 1.0 6.98e+03 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  4  0  0  0   0  4  0  0  0   380
> MatMult               30 1.0 7.5340e-05 1.0 8.07e+04 1.0 0.0e+00 0.0e+00
> 0.0e+00  0 42  0  0  0   0 42  0  0  0  1071
>
> Given the matrix size and the number of nonzeros, vector size of my
> solution, and the number of calls to each of these vec ops, I can estimate
> the total bytes transferred (loads/stores)?
>

Yes, exactly.


> For multiple processors, do I calculate the local or global sizes? If I
> calculate the local sizes, then do I need to do an MPI_Allreduce (with
> mpi_sum) of the TBT across all processors just like with total flops?
>

Just do this analysis locally I think. Dealing with reductions adds another
level of complexity.


> Do these vec/mat ops account for Dirichlet constraints? That is, should
> the global/local size include those constraints?
>

Hopefully the Dirichlet constraints are not significant. Certainly in the
limit of large N they drop out. The solver operations
are in the global space, and the assembly operations are in the local space.


> Also, is there a way to extract the event count for these operations
> besides dumping -log_summary each time?
>

Yes, using the PetscStageLog and PetscEventLog interfaces. Here is an
example from dtfe.c:

  PetscStageLog     stageLog;
  PetscEventPerfLog eventLog = NULL;
  PetscInt          stage;
  PetscErrorCode    ierr;

  PetscFunctionBegin;
  ierr = PetscLogGetStageLog(&stageLog);CHKERRQ(ierr);
  ierr = PetscStageLogGetCurrent(stageLog, &stage);CHKERRQ(ierr);
  ierr = PetscStageLogGetEventPerfLog(stageLog, stage,
&eventLog);CHKERRQ(ierr);
    /* Log performance info */
  eventLog->eventInfo[ocl->residualEvent].count++;
  eventLog->eventInfo[ocl->residualEvent].time  += time;
  eventLog->eventInfo[ocl->residualEvent].flops += flops;


  Thanks,

     Matt


> Thanks,
> Justin
>
> On Wed, May 6, 2015 at 1:38 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, May 6, 2015 at 1:28 PM, Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> Jed,
>>>
>>> I am working with anisotropic diffusion and most standard numerical
>>> formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
>>> discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
>>> on this. What we have seen people do is simply "ignore" or chop off these
>>> values but to us that is a complete and utter abomination. My goal here is
>>> to show that our proposed methodologies work by leveraging on the
>>> capabilities within PETSc and TAO and to also show how computationally
>>> expensive it is compared to solving the same problem using the standard
>>> Galerkin method.
>>>
>>> Matt,
>>>
>>> Okay, so then I guess I still have questions regarding how to obtain the
>>> bytes. How exactly would I count all the number of Vecs and their
>>> respective sizes, because it seems all the DMPlex related functions create
>>> many vectors. Or do I only count the DM created vectors used for my
>>> solution vector, residual, lower/upper bound, optimization routines, etc?
>>>
>>
>> This is laborious. You would build up from the small stuff. So a Krylov
>> solver have MatMult, for which there is an analysis in the paper with
>> Dinesh/Bill/Barry/David, and
>> Vec ops which are easy. This is a lot of counting, especially if you have
>> a TAO solver in there. I would make sure you really care.
>>
>>
>>> And when you say "forget about small stuff", does that include all the
>>> DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
>>> all the jazz that goes on within the FE/discretization routines?
>>>
>>
>> Yep.
>>
>>
>>> Lastly, for a Matrix, wouldn't I just get the number of bytes from the
>>> memory usage section in -log_summary?
>>>
>>
>> That is a good way. You can also ask MatInfo how many nonzeros the matrix
>> has.
>>
>>    Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> I suppose I have two objectives that I think are achievable within
>>>>> PETSc means:
>>>>>
>>>>> 1) How much wall-clock time can be reduced as you increase the number
>>>>> of processors. I have strong-scaling and parallel efficiency metrics that
>>>>> convey this.
>>>>>
>>>>> 2) The "optimal" problem size for these two methods/solvers are. What
>>>>> I mean by this is, at what point do I achieve the maximum FLOPS/s. If
>>>>> starting off with a really small problem then this metric should increase
>>>>> with problem size. My hypothesis is that as problem size increases, the
>>>>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>>>>> accessing main memory, etc) to performing work also increases, and the
>>>>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>>>>> this context simply means the highest possible FLOPS/s.
>>>>>
>>>>> Does that make sense and/or is "interesting" enough?
>>>>>
>>>>
>>>> I think 2) is not really that interesting because
>>>>
>>>>   a) it is so easily gamed. Just stick in high flop count operations,
>>>> like DGEMM.
>>>>
>>>>   b) Time really matters to people who run the code, but flops never do.
>>>>
>>>>   c) Floating point performance is not your limiting factor for time
>>>>
>>>> I think it would be much more interesting, and no more work to
>>>>
>>>>   a) Model the flop/byte \beta ratio simply
>>>>
>>>>   b) Report how close you get to the max performance given \beta on
>>>> your machine
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Justin
>>>>>
>>>>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>>> Justin Chang <jychang48 at gmail.com> writes:
>>>>>> > I already have speedup/strong scaling results that essentially
>>>>>> depict the
>>>>>> > difference between the KSPSolve() and TaoSolve(). However, I have
>>>>>> been told
>>>>>> > by someone that strong-scaling isn't enough - that I should somehow
>>>>>> include
>>>>>> > something to show the "efficiency" of these two methodologies.
>>>>>>
>>>>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>>>>>> where both get the right answer and vary a parameter to get to the
>>>>>> case
>>>>>> where one fails?  Then you can look at efficiency for a given accuracy
>>>>>> (and you might have to refine the grid differently) as you vary the
>>>>>> parameter.
>>>>>>
>>>>>> It's really hard to demonstrate that an implicit solver is optimal in
>>>>>> terms of mathematical convergence rate.  Improvements there can dwarf
>>>>>> any differences in implementation efficiency.
>>>>>>
>>>>>> > That is, how much of the wall-clock time reported by these two very
>>>>>> > different solvers is spent doing useful work.
>>>>>> >
>>>>>> > Is such an "efficiency" metric necessary to report in addition to
>>>>>> > strong-scaling results? The overall computational framework is the
>>>>>> same for
>>>>>> > both problems, the only difference being one uses a linear solver
>>>>>> and the
>>>>>> > other uses an optimization solver. My first thought was to use PAPI
>>>>>> to
>>>>>> > include hardware counters, but these are notoriously inaccurate.
>>>>>> Then I
>>>>>> > thought about simply reporting the manual FLOPS and FLOPS/s via
>>>>>> PETSc, but
>>>>>> > these metrics ignore memory bandwidth. And so here I am looking at
>>>>>> the idea
>>>>>> > of implementing the Roofline model, but now I am wondering if any
>>>>>> of this
>>>>>> > is worth the trouble.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/285befc0/attachment-0001.html>

From gideon.simpson at gmail.com  Wed May  6 17:24:43 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Wed, 6 May 2015 18:24:43 -0400
Subject: [petsc-users] DM Vector get vs. create
Message-ID: <748AE6CC-02F9-4AF3-8E62-739FF00B7D26@gmail.com>

What are the relative merits of DMGetGlobalVector and DMCreateGlobalVector, and the analogous question for the Local Vectors?  Are there scenarios where one should be used over the other?

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/dc18b2cc/attachment.html>

From bsmith at mcs.anl.gov  Wed May  6 17:32:09 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 6 May 2015 17:32:09 -0500
Subject: [petsc-users] DM Vector get vs. create
In-Reply-To: <748AE6CC-02F9-4AF3-8E62-739FF00B7D26@gmail.com>
References: <748AE6CC-02F9-4AF3-8E62-739FF00B7D26@gmail.com>
Message-ID: <4F8A0C3C-A9D1-4479-8C2C-76D0422FD934@mcs.anl.gov>


> On May 6, 2015, at 5:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> What are the relative merits of DMGetGlobalVector and DMCreateGlobalVector, and the analogous question for the Local Vectors?  Are there scenarios where one should be used over the other?

   Get are intended for obtaining work vectors that one intends to restore soon, likely in the same function that did the get. Create is for long living vectors that you are likely passing around to solvers, subroutines etc.

    Explanation: sometimes a subroutine needs some work vectors. For example often a PETSc SNESFormFunction will do a DMGlobalToLocalBegin/End() at the beginning to allow access of ghost values. That local vector is only needed in that subroutine so one should use the get.


> 
> -gideon
> 


From rtm at utk.edu  Wed May  6 18:49:54 2015
From: rtm at utk.edu (Richard Mills)
Date: Wed, 6 May 2015 16:49:54 -0700
Subject: [petsc-users] DM Vector get vs. create
In-Reply-To: <4F8A0C3C-A9D1-4479-8C2C-76D0422FD934@mcs.anl.gov>
References: <748AE6CC-02F9-4AF3-8E62-739FF00B7D26@gmail.com>
	<4F8A0C3C-A9D1-4479-8C2C-76D0422FD934@mcs.anl.gov>
Message-ID: <CAAZcC2d=a39XbPQp8cgYKTxtht7b5_C+d5CA2OqLGqFWvWS-tA@mail.gmail.com>

Barry,

Perhaps it would be good to put your explanation into the manual pages for
these routines?  The difference between the two is not obvious without
looking at examples.

--Richard

On Wed, May 6, 2015 at 3:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 6, 2015, at 5:24 PM, Gideon Simpson <gideon.simpson at gmail.com>
> wrote:
> >
> > What are the relative merits of DMGetGlobalVector and
> DMCreateGlobalVector, and the analogous question for the Local Vectors?
> Are there scenarios where one should be used over the other?
>
>    Get are intended for obtaining work vectors that one intends to restore
> soon, likely in the same function that did the get. Create is for long
> living vectors that you are likely passing around to solvers, subroutines
> etc.
>
>     Explanation: sometimes a subroutine needs some work vectors. For
> example often a PETSc SNESFormFunction will do a DMGlobalToLocalBegin/End()
> at the beginning to allow access of ghost values. That local vector is only
> needed in that subroutine so one should use the get.
>
>
> >
> > -gideon
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150506/8e861079/attachment.html>

From jychang48 at gmail.com  Thu May  7 08:26:39 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 7 May 2015 08:26:39 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GmgHe_XfqV0usFXgKCgfoDGLU-QE0+cqiD8hj0jg8b=7A@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
	<CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>
	<CAP2=TMhBcTriQVCGtKmW=cj=hMhnBgrYthKMbAks6R1xNUz99Q@mail.gmail.com>
	<CAMYG4GmgHe_XfqV0usFXgKCgfoDGLU-QE0+cqiD8hj0jg8b=7A@mail.gmail.com>
Message-ID: <CAP2=TMiaceQLBprVzHKgnBwo3pkcKV7wYFbqGBjysEu-kxkqng@mail.gmail.com>

Matt again thank you very much for all your help, last question(s) though.

I am not quite sure I follow your second comment. Do you mean that this
whole analysis should be done on one processor (in sequential mode)? Or do
you mean that calculating the TBT for these vec and matmult ops should
assume the local size of the vector/matrix? When I said MPI_Allreduce, i
meant collecting everything after calculating the local TBT for each
processor. Most of these vec ops' calculations are local so it shouldn't be
too much trouble, but for things like matmult and vecnorm, would it be
quite convoluted because of interprocessor communication?

Thanks,
Justin

On Wed, May 6, 2015 at 2:34 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 6, 2015 at 2:28 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> So basically I just need these types of operations:
>>
>> VecTDot               60 1.0 1.6928e-05 1.0 2.69e+04 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1591
>> VecNorm               31 1.0 9.0599e-06 1.0 1.39e+04 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0  7  0  0  0   0  7  0  0  0  1536
>> VecCopy                3 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecSet                36 1.0 9.4175e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>> VecAXPY               60 1.0 1.7166e-05 1.0 2.70e+04 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1573
>> VecAYPX               29 1.0 1.9312e-05 1.0 1.30e+04 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0  7  0  0  0   0  7  0  0  0   676
>> VecWAXPY               1 1.0 1.9073e-06 1.0 2.25e+02 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0  0  0  0  0   0  0  0  0  0   118
>> VecPointwiseMult      31 1.0 1.8358e-05 1.0 6.98e+03 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0  4  0  0  0   0  4  0  0  0   380
>> MatMult               30 1.0 7.5340e-05 1.0 8.07e+04 1.0 0.0e+00 0.0e+00
>> 0.0e+00  0 42  0  0  0   0 42  0  0  0  1071
>>
>> Given the matrix size and the number of nonzeros, vector size of my
>> solution, and the number of calls to each of these vec ops, I can estimate
>> the total bytes transferred (loads/stores)?
>>
>
> Yes, exactly.
>
>
>> For multiple processors, do I calculate the local or global sizes? If I
>> calculate the local sizes, then do I need to do an MPI_Allreduce (with
>> mpi_sum) of the TBT across all processors just like with total flops?
>>
>
> Just do this analysis locally I think. Dealing with reductions adds
> another level of complexity.
>
>
>> Do these vec/mat ops account for Dirichlet constraints? That is, should
>> the global/local size include those constraints?
>>
>
> Hopefully the Dirichlet constraints are not significant. Certainly in the
> limit of large N they drop out. The solver operations
> are in the global space, and the assembly operations are in the local
> space.
>
>
>> Also, is there a way to extract the event count for these operations
>> besides dumping -log_summary each time?
>>
>
> Yes, using the PetscStageLog and PetscEventLog interfaces. Here is an
> example from dtfe.c:
>
>   PetscStageLog     stageLog;
>   PetscEventPerfLog eventLog = NULL;
>   PetscInt          stage;
>   PetscErrorCode    ierr;
>
>   PetscFunctionBegin;
>   ierr = PetscLogGetStageLog(&stageLog);CHKERRQ(ierr);
>   ierr = PetscStageLogGetCurrent(stageLog, &stage);CHKERRQ(ierr);
>   ierr = PetscStageLogGetEventPerfLog(stageLog, stage,
> &eventLog);CHKERRQ(ierr);
>     /* Log performance info */
>   eventLog->eventInfo[ocl->residualEvent].count++;
>   eventLog->eventInfo[ocl->residualEvent].time  += time;
>   eventLog->eventInfo[ocl->residualEvent].flops += flops;
>
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Justin
>>
>> On Wed, May 6, 2015 at 1:38 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Wed, May 6, 2015 at 1:28 PM, Justin Chang <jychang48 at gmail.com>
>>> wrote:
>>>
>>>> Jed,
>>>>
>>>> I am working with anisotropic diffusion and most standard numerical
>>>> formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
>>>> discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
>>>> on this. What we have seen people do is simply "ignore" or chop off these
>>>> values but to us that is a complete and utter abomination. My goal here is
>>>> to show that our proposed methodologies work by leveraging on the
>>>> capabilities within PETSc and TAO and to also show how computationally
>>>> expensive it is compared to solving the same problem using the standard
>>>> Galerkin method.
>>>>
>>>> Matt,
>>>>
>>>> Okay, so then I guess I still have questions regarding how to obtain
>>>> the bytes. How exactly would I count all the number of Vecs and their
>>>> respective sizes, because it seems all the DMPlex related functions create
>>>> many vectors. Or do I only count the DM created vectors used for my
>>>> solution vector, residual, lower/upper bound, optimization routines, etc?
>>>>
>>>
>>> This is laborious. You would build up from the small stuff. So a Krylov
>>> solver have MatMult, for which there is an analysis in the paper with
>>> Dinesh/Bill/Barry/David, and
>>> Vec ops which are easy. This is a lot of counting, especially if you
>>> have a TAO solver in there. I would make sure you really care.
>>>
>>>
>>>> And when you say "forget about small stuff", does that include all the
>>>> DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
>>>> all the jazz that goes on within the FE/discretization routines?
>>>>
>>>
>>> Yep.
>>>
>>>
>>>> Lastly, for a Matrix, wouldn't I just get the number of bytes from the
>>>> memory usage section in -log_summary?
>>>>
>>>
>>> That is a good way. You can also ask MatInfo how many nonzeros the
>>> matrix has.
>>>
>>>    Matt
>>>
>>>
>>>> Thanks,
>>>> Justin
>>>>
>>>> On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I suppose I have two objectives that I think are achievable within
>>>>>> PETSc means:
>>>>>>
>>>>>> 1) How much wall-clock time can be reduced as you increase the number
>>>>>> of processors. I have strong-scaling and parallel efficiency metrics that
>>>>>> convey this.
>>>>>>
>>>>>> 2) The "optimal" problem size for these two methods/solvers are. What
>>>>>> I mean by this is, at what point do I achieve the maximum FLOPS/s. If
>>>>>> starting off with a really small problem then this metric should increase
>>>>>> with problem size. My hypothesis is that as problem size increases, the
>>>>>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>>>>>> accessing main memory, etc) to performing work also increases, and the
>>>>>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>>>>>> this context simply means the highest possible FLOPS/s.
>>>>>>
>>>>>> Does that make sense and/or is "interesting" enough?
>>>>>>
>>>>>
>>>>> I think 2) is not really that interesting because
>>>>>
>>>>>   a) it is so easily gamed. Just stick in high flop count operations,
>>>>> like DGEMM.
>>>>>
>>>>>   b) Time really matters to people who run the code, but flops never
>>>>> do.
>>>>>
>>>>>   c) Floating point performance is not your limiting factor for time
>>>>>
>>>>> I think it would be much more interesting, and no more work to
>>>>>
>>>>>   a) Model the flop/byte \beta ratio simply
>>>>>
>>>>>   b) Report how close you get to the max performance given \beta on
>>>>> your machine
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Justin
>>>>>>
>>>>>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>>
>>>>>>> Justin Chang <jychang48 at gmail.com> writes:
>>>>>>> > I already have speedup/strong scaling results that essentially
>>>>>>> depict the
>>>>>>> > difference between the KSPSolve() and TaoSolve(). However, I have
>>>>>>> been told
>>>>>>> > by someone that strong-scaling isn't enough - that I should
>>>>>>> somehow include
>>>>>>> > something to show the "efficiency" of these two methodologies.
>>>>>>>
>>>>>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a problem
>>>>>>> where both get the right answer and vary a parameter to get to the
>>>>>>> case
>>>>>>> where one fails?  Then you can look at efficiency for a given
>>>>>>> accuracy
>>>>>>> (and you might have to refine the grid differently) as you vary the
>>>>>>> parameter.
>>>>>>>
>>>>>>> It's really hard to demonstrate that an implicit solver is optimal in
>>>>>>> terms of mathematical convergence rate.  Improvements there can dwarf
>>>>>>> any differences in implementation efficiency.
>>>>>>>
>>>>>>> > That is, how much of the wall-clock time reported by these two very
>>>>>>> > different solvers is spent doing useful work.
>>>>>>> >
>>>>>>> > Is such an "efficiency" metric necessary to report in addition to
>>>>>>> > strong-scaling results? The overall computational framework is the
>>>>>>> same for
>>>>>>> > both problems, the only difference being one uses a linear solver
>>>>>>> and the
>>>>>>> > other uses an optimization solver. My first thought was to use
>>>>>>> PAPI to
>>>>>>> > include hardware counters, but these are notoriously inaccurate.
>>>>>>> Then I
>>>>>>> > thought about simply reporting the manual FLOPS and FLOPS/s via
>>>>>>> PETSc, but
>>>>>>> > these metrics ignore memory bandwidth. And so here I am looking at
>>>>>>> the idea
>>>>>>> > of implementing the Roofline model, but now I am wondering if any
>>>>>>> of this
>>>>>>> > is worth the trouble.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/341b0a2f/attachment-0001.html>

From jed at jedbrown.org  Thu May  7 08:35:23 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 07 May 2015 09:35:23 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
Message-ID: <876184fr38.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> I think it would be much more interesting, and no more work to
>
>   a) Model the flop/byte \beta ratio simply
>
>   b) Report how close you get to the max performance given \beta on your
> machine

This is also easily gamed; just use high-memory data structures, extra
STREAM copies, etc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/8a689a42/attachment.pgp>

From knepley at gmail.com  Thu May  7 08:37:57 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 08:37:57 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMiaceQLBprVzHKgnBwo3pkcKV7wYFbqGBjysEu-kxkqng@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<CAP2=TMh2bgv=R56oPibCS48DNvJsOBA+Bx-rD5oK4fiUcT0tfw@mail.gmail.com>
	<CAMYG4GmXcN=YzAHu1o_dYawLC74KcPXRvZ-7pn7kHmfgsUAScQ@mail.gmail.com>
	<CAP2=TMhBcTriQVCGtKmW=cj=hMhnBgrYthKMbAks6R1xNUz99Q@mail.gmail.com>
	<CAMYG4GmgHe_XfqV0usFXgKCgfoDGLU-QE0+cqiD8hj0jg8b=7A@mail.gmail.com>
	<CAP2=TMiaceQLBprVzHKgnBwo3pkcKV7wYFbqGBjysEu-kxkqng@mail.gmail.com>
Message-ID: <CAMYG4GnRJuE6WoF8FfjrBgEm6hViaCUtQZ71RRNDj00QkikjeQ@mail.gmail.com>

On Thu, May 7, 2015 at 8:26 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Matt again thank you very much for all your help, last question(s) though.
>
> I am not quite sure I follow your second comment. Do you mean that this
> whole analysis should be done on one processor (in sequential mode)? Or do
> you mean that calculating the TBT for these vec and matmult ops should
> assume the local size of the vector/matrix? When I said MPI_Allreduce, i
> meant collecting everything after calculating the local TBT for each
> processor. Most of these vec ops' calculations are local so it shouldn't be
> too much trouble, but for things like matmult and vecnorm, would it be
> quite convoluted because of interprocessor communication?
>

I meant that I would do the analysis for one process and assume that is
generalized smoothly.

  Matt


> Thanks,
> Justin
>
> On Wed, May 6, 2015 at 2:34 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, May 6, 2015 at 2:28 PM, Justin Chang <jychang48 at gmail.com> wrote:
>>
>>> So basically I just need these types of operations:
>>>
>>> VecTDot               60 1.0 1.6928e-05 1.0 2.69e+04 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1591
>>> VecNorm               31 1.0 9.0599e-06 1.0 1.39e+04 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0  7  0  0  0   0  7  0  0  0  1536
>>> VecCopy                3 1.0 9.5367e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecSet                36 1.0 9.4175e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>> 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>>> VecAXPY               60 1.0 1.7166e-05 1.0 2.70e+04 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0 14  0  0  0   0 14  0  0  0  1573
>>> VecAYPX               29 1.0 1.9312e-05 1.0 1.30e+04 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0  7  0  0  0   0  7  0  0  0   676
>>> VecWAXPY               1 1.0 1.9073e-06 1.0 2.25e+02 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0  0  0  0  0   0  0  0  0  0   118
>>> VecPointwiseMult      31 1.0 1.8358e-05 1.0 6.98e+03 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0  4  0  0  0   0  4  0  0  0   380
>>> MatMult               30 1.0 7.5340e-05 1.0 8.07e+04 1.0 0.0e+00 0.0e+00
>>> 0.0e+00  0 42  0  0  0   0 42  0  0  0  1071
>>>
>>> Given the matrix size and the number of nonzeros, vector size of my
>>> solution, and the number of calls to each of these vec ops, I can estimate
>>> the total bytes transferred (loads/stores)?
>>>
>>
>> Yes, exactly.
>>
>>
>>> For multiple processors, do I calculate the local or global sizes? If I
>>> calculate the local sizes, then do I need to do an MPI_Allreduce (with
>>> mpi_sum) of the TBT across all processors just like with total flops?
>>>
>>
>> Just do this analysis locally I think. Dealing with reductions adds
>> another level of complexity.
>>
>>
>>> Do these vec/mat ops account for Dirichlet constraints? That is, should
>>> the global/local size include those constraints?
>>>
>>
>> Hopefully the Dirichlet constraints are not significant. Certainly in the
>> limit of large N they drop out. The solver operations
>> are in the global space, and the assembly operations are in the local
>> space.
>>
>>
>>> Also, is there a way to extract the event count for these operations
>>> besides dumping -log_summary each time?
>>>
>>
>> Yes, using the PetscStageLog and PetscEventLog interfaces. Here is an
>> example from dtfe.c:
>>
>>   PetscStageLog     stageLog;
>>   PetscEventPerfLog eventLog = NULL;
>>   PetscInt          stage;
>>   PetscErrorCode    ierr;
>>
>>   PetscFunctionBegin;
>>   ierr = PetscLogGetStageLog(&stageLog);CHKERRQ(ierr);
>>   ierr = PetscStageLogGetCurrent(stageLog, &stage);CHKERRQ(ierr);
>>   ierr = PetscStageLogGetEventPerfLog(stageLog, stage,
>> &eventLog);CHKERRQ(ierr);
>>     /* Log performance info */
>>   eventLog->eventInfo[ocl->residualEvent].count++;
>>   eventLog->eventInfo[ocl->residualEvent].time  += time;
>>   eventLog->eventInfo[ocl->residualEvent].flops += flops;
>>
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>> On Wed, May 6, 2015 at 1:38 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Wed, May 6, 2015 at 1:28 PM, Justin Chang <jychang48 at gmail.com>
>>>> wrote:
>>>>
>>>>> Jed,
>>>>>
>>>>> I am working with anisotropic diffusion and most standard numerical
>>>>> formulations (e.g., FEM, FVM, etc.) are "wrong" because they violate the
>>>>> discrete maximum principle, see Nakshatrala & Valocci (JCP 2009) for more
>>>>> on this. What we have seen people do is simply "ignore" or chop off these
>>>>> values but to us that is a complete and utter abomination. My goal here is
>>>>> to show that our proposed methodologies work by leveraging on the
>>>>> capabilities within PETSc and TAO and to also show how computationally
>>>>> expensive it is compared to solving the same problem using the standard
>>>>> Galerkin method.
>>>>>
>>>>> Matt,
>>>>>
>>>>> Okay, so then I guess I still have questions regarding how to obtain
>>>>> the bytes. How exactly would I count all the number of Vecs and their
>>>>> respective sizes, because it seems all the DMPlex related functions create
>>>>> many vectors. Or do I only count the DM created vectors used for my
>>>>> solution vector, residual, lower/upper bound, optimization routines, etc?
>>>>>
>>>>
>>>> This is laborious. You would build up from the small stuff. So a Krylov
>>>> solver have MatMult, for which there is an analysis in the paper with
>>>> Dinesh/Bill/Barry/David, and
>>>> Vec ops which are easy. This is a lot of counting, especially if you
>>>> have a TAO solver in there. I would make sure you really care.
>>>>
>>>>
>>>>> And when you say "forget about small stuff", does that include all the
>>>>> DMPlex creation routines, PetscMalloc'ed arrays, pointwise functions, and
>>>>> all the jazz that goes on within the FE/discretization routines?
>>>>>
>>>>
>>>> Yep.
>>>>
>>>>
>>>>> Lastly, for a Matrix, wouldn't I just get the number of bytes from the
>>>>> memory usage section in -log_summary?
>>>>>
>>>>
>>>> That is a good way. You can also ask MatInfo how many nonzeros the
>>>> matrix has.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Justin
>>>>>
>>>>> On Wed, May 6, 2015 at 11:48 AM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Wed, May 6, 2015 at 11:41 AM, Justin Chang <jychang48 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I suppose I have two objectives that I think are achievable within
>>>>>>> PETSc means:
>>>>>>>
>>>>>>> 1) How much wall-clock time can be reduced as you increase the
>>>>>>> number of processors. I have strong-scaling and parallel efficiency metrics
>>>>>>> that convey this.
>>>>>>>
>>>>>>> 2) The "optimal" problem size for these two methods/solvers are.
>>>>>>> What I mean by this is, at what point do I achieve the maximum FLOPS/s. If
>>>>>>> starting off with a really small problem then this metric should increase
>>>>>>> with problem size. My hypothesis is that as problem size increases, the
>>>>>>> ratio of wall-clock time spent in idle (e.g., waiting for cache to free up,
>>>>>>> accessing main memory, etc) to performing work also increases, and the
>>>>>>> reported FLOPS/s should start decreasing at some point. "Efficiency" in
>>>>>>> this context simply means the highest possible FLOPS/s.
>>>>>>>
>>>>>>> Does that make sense and/or is "interesting" enough?
>>>>>>>
>>>>>>
>>>>>> I think 2) is not really that interesting because
>>>>>>
>>>>>>   a) it is so easily gamed. Just stick in high flop count operations,
>>>>>> like DGEMM.
>>>>>>
>>>>>>   b) Time really matters to people who run the code, but flops never
>>>>>> do.
>>>>>>
>>>>>>   c) Floating point performance is not your limiting factor for time
>>>>>>
>>>>>> I think it would be much more interesting, and no more work to
>>>>>>
>>>>>>   a) Model the flop/byte \beta ratio simply
>>>>>>
>>>>>>   b) Report how close you get to the max performance given \beta on
>>>>>> your machine
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Justin
>>>>>>>
>>>>>>> On Wed, May 6, 2015 at 11:28 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>>>
>>>>>>>> Justin Chang <jychang48 at gmail.com> writes:
>>>>>>>> > I already have speedup/strong scaling results that essentially
>>>>>>>> depict the
>>>>>>>> > difference between the KSPSolve() and TaoSolve(). However, I have
>>>>>>>> been told
>>>>>>>> > by someone that strong-scaling isn't enough - that I should
>>>>>>>> somehow include
>>>>>>>> > something to show the "efficiency" of these two methodologies.
>>>>>>>>
>>>>>>>> "Efficiency" is irrelevant if one is wrong.  Can you set up a
>>>>>>>> problem
>>>>>>>> where both get the right answer and vary a parameter to get to the
>>>>>>>> case
>>>>>>>> where one fails?  Then you can look at efficiency for a given
>>>>>>>> accuracy
>>>>>>>> (and you might have to refine the grid differently) as you vary the
>>>>>>>> parameter.
>>>>>>>>
>>>>>>>> It's really hard to demonstrate that an implicit solver is optimal
>>>>>>>> in
>>>>>>>> terms of mathematical convergence rate.  Improvements there can
>>>>>>>> dwarf
>>>>>>>> any differences in implementation efficiency.
>>>>>>>>
>>>>>>>> > That is, how much of the wall-clock time reported by these two
>>>>>>>> very
>>>>>>>> > different solvers is spent doing useful work.
>>>>>>>> >
>>>>>>>> > Is such an "efficiency" metric necessary to report in addition to
>>>>>>>> > strong-scaling results? The overall computational framework is
>>>>>>>> the same for
>>>>>>>> > both problems, the only difference being one uses a linear solver
>>>>>>>> and the
>>>>>>>> > other uses an optimization solver. My first thought was to use
>>>>>>>> PAPI to
>>>>>>>> > include hardware counters, but these are notoriously inaccurate.
>>>>>>>> Then I
>>>>>>>> > thought about simply reporting the manual FLOPS and FLOPS/s via
>>>>>>>> PETSc, but
>>>>>>>> > these metrics ignore memory bandwidth. And so here I am looking
>>>>>>>> at the idea
>>>>>>>> > of implementing the Roofline model, but now I am wondering if any
>>>>>>>> of this
>>>>>>>> > is worth the trouble.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/8e5d853c/attachment.html>

From knepley at gmail.com  Thu May  7 08:38:38 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 08:38:38 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <876184fr38.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
Message-ID: <CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>

On Thu, May 7, 2015 at 8:35 AM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > I think it would be much more interesting, and no more work to
> >
> >   a) Model the flop/byte \beta ratio simply
> >
> >   b) Report how close you get to the max performance given \beta on your
> > machine
>
> This is also easily gamed; just use high-memory data structures, extra
> STREAM copies, etc.
>

I completely agree that all the performance measures can be gamed, which is
why
its important to always show time to solution as well.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/29a58432/attachment-0001.html>

From jychang48 at gmail.com  Thu May  7 09:23:30 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 7 May 2015 09:23:30 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
Message-ID: <CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>

So to summarize, if I understand everything, I should do the following:

1) calculate the flop/byte ratio for various problem sizes and solver
methods on one process and:

a) use the ratio to estimate the ideal flops/s based on the stream bw.
Compare this to the measured flops/s from my Petsc program?

b) show the associated wall-clock times.

c) warn that this model is uncached and merely gives a rough estimate of
the performance.

2) Do a strong-scaling/speed up study to illustrate how these problems
scale across multiple processes. Optionally see what the optimal number of
processes is required for various problem sizes and solvers.

Am I missing anything?

Thanks,
Justin

On Thursday, May 7, 2015, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, May 7, 2015 at 8:35 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>> > I think it would be much more interesting, and no more work to
>> >
>> >   a) Model the flop/byte \beta ratio simply
>> >
>> >   b) Report how close you get to the max performance given \beta on your
>> > machine
>>
>> This is also easily gamed; just use high-memory data structures, extra
>> STREAM copies, etc.
>>
>
> I completely agree that all the performance measures can be gamed, which
> is why
> its important to always show time to solution as well.
>
>    Matt
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/08fc4855/attachment.html>

From knepley at gmail.com  Thu May  7 09:29:08 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 09:29:08 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
Message-ID: <CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>

On Thu, May 7, 2015 at 9:23 AM, Justin Chang <jychang48 at gmail.com> wrote:

> So to summarize, if I understand everything, I should do the following:
>
> 1) calculate the flop/byte ratio for various problem sizes and solver
> methods on one process and:
>

Should be roughly invariant to problem size.


> a) use the ratio to estimate the ideal flops/s based on the stream bw.
> Compare this to the measured flops/s from my Petsc program?
>

Yes


> b) show the associated wall-clock times.
>

Your strong scaling plots are fine here.


> c) warn that this model is uncached and merely gives a rough estimate of
> the performance.
>

Yes, I would say "does not deal with cache effects".


> 2) Do a strong-scaling/speed up study to illustrate how these problems
> scale across multiple processes. Optionally see what the optimal number of
> processes is required for various problem sizes and solvers.
>
> Am I missing anything?
>

Sounds good to me.

   Matt


> Thanks,
> Justin
>
>
> On Thursday, May 7, 2015, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, May 7, 2015 at 8:35 AM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Matthew Knepley <knepley at gmail.com> writes:
>>> > I think it would be much more interesting, and no more work to
>>> >
>>> >   a) Model the flop/byte \beta ratio simply
>>> >
>>> >   b) Report how close you get to the max performance given \beta on
>>> your
>>> > machine
>>>
>>> This is also easily gamed; just use high-memory data structures, extra
>>> STREAM copies, etc.
>>>
>>
>> I completely agree that all the performance measures can be gamed, which
>> is why
>> its important to always show time to solution as well.
>>
>>    Matt
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/e0c2f1e6/attachment.html>

From hypre-support at llnl.gov  Thu May  7 13:15:40 2015
From: hypre-support at llnl.gov (Ulrike Yang hypre Tracker)
Date: Thu, 07 May 2015 18:15:40 -0000
Subject: [petsc-users] [issue1287] divide by zero coming in hypre
Message-ID: <1431022540.51.0.895981966755.issue1287@llnl.gov>


Ulrike Yang <umyang at llnl.gov> added the comment:

fixed this and pushed it to the master
Ulrike

----------
assignedto:  -> ulrikey
priority:  -> bug
status: chatting -> done-cbb

_____________________________________________
hypre Issue Tracker <hypre-support at llnl.gov>
<http://tux273.llnl.gov:8070/hypre/issue1287>
_____________________________________________

From jed at jedbrown.org  Thu May  7 13:25:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 07 May 2015 14:25:12 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
Message-ID: <87mw1gdz3r.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, May 7, 2015 at 9:23 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> So to summarize, if I understand everything, I should do the following:
>>
>> 1) calculate the flop/byte ratio for various problem sizes and solver
>> methods on one process and:

It is impossible to define "arithmetic intensity" (flops/byte) without
selecting a cache model.  And my "cache model", I include register
reuse.  Assuming no reuse at all is not even close to representative
(off by orders of magnitude).

> Should be roughly invariant to problem size.

Except (a) insofar as it depends on cache sizes and (b) the algorithm
characteristics depend on data size (e.g., the cost of supernode
factorization for sparse direct solvers scales superlinearly, so
arithmetic intensity will drift as you increase the problem size).


You can see how we compare "pessimal" versus "perfect" caching here:

http://59A2.org/na/MayBrownLePourhiet-pTatin3d-2014.pdf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/05e2b37f/attachment.pgp>

From knepley at gmail.com  Thu May  7 13:28:28 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 13:28:28 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87mw1gdz3r.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org>
Message-ID: <CAMYG4Gkj5U10a=VRJuMGR_ndkyRT6OLCGaL5bp3sOXY13RakfQ@mail.gmail.com>

On Thu, May 7, 2015 at 1:25 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Thu, May 7, 2015 at 9:23 AM, Justin Chang <jychang48 at gmail.com>
> wrote:
> >
> >> So to summarize, if I understand everything, I should do the following:
> >>
> >> 1) calculate the flop/byte ratio for various problem sizes and solver
> >> methods on one process and:
>
> It is impossible to define "arithmetic intensity" (flops/byte) without
> selecting a cache model.  And my "cache model", I include register
> reuse.  Assuming no reuse at all is not even close to representative
> (off by orders of magnitude).


You are making assumptions about the algorithm here. Pure streaming
computations,
like VecAXPY do not depend on the cache model.

  Matt


>
> > Should be roughly invariant to problem size.
>
> Except (a) insofar as it depends on cache sizes and (b) the algorithm
> characteristics depend on data size (e.g., the cost of supernode
> factorization for sparse direct solvers scales superlinearly, so
> arithmetic intensity will drift as you increase the problem size).
>
>
> You can see how we compare "pessimal" versus "perfect" caching here:
>
> http://59A2.org/na/MayBrownLePourhiet-pTatin3d-2014.pdf
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/569d0fc7/attachment.html>

From bsmith at mcs.anl.gov  Thu May  7 13:31:11 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 7 May 2015 13:31:11 -0500
Subject: [petsc-users] [issue1287] divide by zero coming in hypre
In-Reply-To: <1431022540.51.0.895981966755.issue1287@llnl.gov>
References: <1431022540.51.0.895981966755.issue1287@llnl.gov>
Message-ID: <35F978AF-E52B-4514-A562-ED394E7BB48C@mcs.anl.gov>


  Ulrike,

    Thanks. Can we get access to master or a tarball for master so we can test it and use it?

   Thanks

   Barry

> On May 7, 2015, at 1:15 PM, Ulrike Yang hypre Tracker <hypre-support at llnl.gov> wrote:
> 
> 
> Ulrike Yang <umyang at llnl.gov> added the comment:
> 
> fixed this and pushed it to the master
> Ulrike
> 
> ----------
> assignedto:  -> ulrikey
> priority:  -> bug
> status: chatting -> done-cbb
> 
> _____________________________________________
> hypre Issue Tracker <hypre-support at llnl.gov>
> <http://tux273.llnl.gov:8070/hypre/issue1287>
> _____________________________________________


From jed at jedbrown.org  Thu May  7 13:32:30 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 07 May 2015 14:32:30 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4Gkj5U10a=VRJuMGR_ndkyRT6OLCGaL5bp3sOXY13RakfQ@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org> <CAMYG4Gk j5U10a=V
	RJuMGR_ndkyRT6OLCGaL5bp3sOXY13RakfQ@mail.gmail.com>
Message-ID: <87k2wkdyrl.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> You are making assumptions about the algorithm here. Pure streaming
> computations, like VecAXPY do not depend on the cache model.

I thought we were talking about Krylov solvers, which includes MatMult.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/d7918475/attachment.pgp>

From knepley at gmail.com  Thu May  7 13:35:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 13:35:10 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87k2wkdyrl.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org>
	<CAMYG4Gkj5U10a=VRJuMGR_ndkyRT6OLCGaL5bp3sOXY13RakfQ@mail.gmail.com>
	<87k2wkdyrl.fsf@jedbrown.org>
Message-ID: <CAMYG4G=6T=dymx=UO=9iVJAG7UNh-_o9=8UU5BfExNVkKzQoHw@mail.gmail.com>

On Thu, May 7, 2015 at 1:32 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > You are making assumptions about the algorithm here. Pure streaming
> > computations, like VecAXPY do not depend on the cache model.
>
> I thought we were talking about Krylov solvers, which includes MatMult.
>

Yes, and in the earlier mail I said to just use the model in Dinesh's paper
so
he did not have to worry about this one.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/f8047f89/attachment.html>

From hypre-support at llnl.gov  Thu May  7 13:36:02 2015
From: hypre-support at llnl.gov (Ulrike Yang hypre Tracker)
Date: Thu, 07 May 2015 18:36:02 -0000
Subject: [petsc-users] [issue1287] divide by zero coming in hypre
In-Reply-To: <35F978AF-E52B-4514-A562-ED394E7BB48C@mcs.anl.gov>
Message-ID: <7D65D8EBE6ADFA42852A360449134C026DC49B86@PRDEXMBX-03.the-lab.llnl.gov>


Ulrike Yang <umyang at llnl.gov> added the comment:

Barry,
I am in the process of putting together a prerelease for one of our customers here. Once I have this I will send it to you also, so you can test it.
Ulrike
________________________________________
From: Barry Smith hypre Tracker [hypre-support at llnl.gov]
Sent: Thursday, May 07, 2015 11:32 AM
To: umyang at llnl.gov
Subject: [issue1287] divide by zero coming in hypre

Barry Smith <bsmith at mcs.anl.gov> added the comment:

Ulrike,

    Thanks. Can we get access to master or a tarball for master so we can test it and use it?

   Thanks

   Barry

> On May 7, 2015, at 1:15 PM, Ulrike Yang hypre Tracker <hypre-support at llnl.gov> wrote:
>
>
> Ulrike Yang <umyang at llnl.gov> added the comment:
>
> fixed this and pushed it to the master
> Ulrike
>
> ----------
> assignedto:  -> ulrikey
> priority:  -> bug
> status: chatting -> done-cbb
>
> _____________________________________________
> hypre Issue Tracker <hypre-support at llnl.gov>
> <http://tux273.llnl.gov:8070/hypre/issue1287>
> _____________________________________________

----------
status: done-cbb -> chatting

_____________________________________________
hypre Issue Tracker <hypre-support at llnl.gov>
<http://tux273.llnl.gov:8070/hypre/issue1287>
_____________________________________________

_____________________________________________
hypre Issue Tracker <hypre-support at llnl.gov>
<http://tux273.llnl.gov:8070/hypre/issue1287>
_____________________________________________

From bsmith at mcs.anl.gov  Thu May  7 13:48:06 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 7 May 2015 13:48:06 -0500
Subject: [petsc-users] [issue1287] divide by zero coming in hypre
In-Reply-To: <7D65D8EBE6ADFA42852A360449134C026DC49B86@PRDEXMBX-03.the-lab.llnl.gov>
References: <7D65D8EBE6ADFA42852A360449134C026DC49B86@PRDEXMBX-03.the-lab.llnl.gov>
Message-ID: <31EFA9DC-599F-4BE3-AAE7-CCAACA5E6196@mcs.anl.gov>


  Great, thanks

> On May 7, 2015, at 1:36 PM, Ulrike Yang hypre Tracker <hypre-support at llnl.gov> wrote:
> 
> 
> Ulrike Yang <umyang at llnl.gov> added the comment:
> 
> Barry,
> I am in the process of putting together a prerelease for one of our customers here. Once I have this I will send it to you also, so you can test it.
> Ulrike
> ________________________________________
> From: Barry Smith hypre Tracker [hypre-support at llnl.gov]
> Sent: Thursday, May 07, 2015 11:32 AM
> To: umyang at llnl.gov
> Subject: [issue1287] divide by zero coming in hypre
> 
> Barry Smith <bsmith at mcs.anl.gov> added the comment:
> 
> Ulrike,
> 
>    Thanks. Can we get access to master or a tarball for master so we can test it and use it?
> 
>   Thanks
> 
>   Barry
> 
>> On May 7, 2015, at 1:15 PM, Ulrike Yang hypre Tracker <hypre-support at llnl.gov> wrote:
>> 
>> 
>> Ulrike Yang <umyang at llnl.gov> added the comment:
>> 
>> fixed this and pushed it to the master
>> Ulrike
>> 
>> ----------
>> assignedto:  -> ulrikey
>> priority:  -> bug
>> status: chatting -> done-cbb
>> 
>> _____________________________________________
>> hypre Issue Tracker <hypre-support at llnl.gov>
>> <http://tux273.llnl.gov:8070/hypre/issue1287>
>> _____________________________________________
> 
> ----------
> status: done-cbb -> chatting
> 
> _____________________________________________
> hypre Issue Tracker <hypre-support at llnl.gov>
> <http://tux273.llnl.gov:8070/hypre/issue1287>
> _____________________________________________
> 
> _____________________________________________
> hypre Issue Tracker <hypre-support at llnl.gov>
> <http://tux273.llnl.gov:8070/hypre/issue1287>
> _____________________________________________


From jed at jedbrown.org  Thu May  7 13:51:08 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 07 May 2015 14:51:08 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAMYG4G=6T=dymx=UO=9iVJAG7UNh-_o9=8UU5BfExNVkKzQoHw@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org> <CAMYG4Gk j5U10a=V
	RJuMGR_ndkyRT6OLCGaL5bp3sOXY13RakfQ@mail.gmail.com>
	<87k2wkdyrl.fsf@jedbrown.org>
	<CAMYG4G=6T=dymx=UO=9iVJAG7UNh-_o9=8UU5BfExNVkKzQoHw@mail.gmail.com>
Message-ID: <87h9rodxwj.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, May 7, 2015 at 1:32 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>> > You are making assumptions about the algorithm here. Pure streaming
>> > computations, like VecAXPY do not depend on the cache model.
>>
>> I thought we were talking about Krylov solvers, which includes MatMult.
>>
>
> Yes, and in the earlier mail I said to just use the model in Dinesh's
> paper so he did not have to worry about this one.

That is a cache model (analogous to "perfect cache" in the pTatin
paper).  So don't claim you have no cache model.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/a77858a3/attachment.pgp>

From bsmith at mcs.anl.gov  Thu May  7 15:25:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 7 May 2015 15:25:00 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
Message-ID: <8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>


  I built the code with gfortran and optimization on my mac and run it on one process. It is spending all its time in the unit_l1_ routine and almost no time in the MatSetValues() in fact if I comment out the MatSetValues() line it runs just as slow as before. How can you know the time is spend in the MatSetValues()? Have you timed it also with the MatSetValues() time commented out?

   Barry

> On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> You are right ! I am using the IRPF90 tool to write my program in FORTRAN.
> 
> To use IRPF90 you can clone the github repository and get the irpf90 binary from the bin directory.
> The code to generate the library that contains UNIT_L1_ can be found at: https://github.com/v1j4y/slepc_version.git
> just clone this repository and do:
> 
> make
> make irpf90.a
> 
> That's it !
> 
> I'm resending you the PETSc files along with the makefile. you'll need the irpf90.a library to make the ex1 executable.
> 
> Hopefully this time it'll work.
> Please let me know if something goes wrong.
> 
> On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Our stupid mail system blocked the attachment because it contained  irpf90.a   Is this a library generated with https://github.com/scemama/irpf90 ?
> You need to email everything that goes into making that library because it is unlikely your .a library will work on my machine. What is this thing and can you just tar up the directory that builds it and I can generate irpf90.a
> 
>   Barry
> 
> > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> >
> > Ok so here goes.
> >
> > I've attached a tarball with a directory which contains the required files to compile ex1.c
> > The UNIT_L1_ subroutine generates the nonzero elements of my hamiltonian given a row, it is contained in the
> > irpf90.a library. The actual fortran code is quite ugly.
> > Please tell me if you are able to compile and run the codes.
> >
> > Thanks a lot,
> >  Vijay
> >
> > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > >
> > > Please let me clarify,
> > >
> > > @Matt:
> > > I've followed the proper manner to rapid assembly.
> > > That is :
> > >   1. preallocation
> > >   2. nonzero ownership
> > > Despite of having done the proper preallocation (as you can see from the output file),
> > > the matrix takes ages to assemble, whereas the same code works for a larger matrix with (100,000,000) elements taking 2 min to assemble !
> > >
> > > @Barry:
> > > I am not touching any file in the code that I attachd.
> > > What i'm doing is that each processor will (according to it's ownership range) get it's nonzero entries
> > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their respective rows.
> > > Again to be clear I"m NOT doing file I/O anywhere.
> >
> >   Oh sorry, where is the fortran code? I assumed the fortran code was reading from a file.
> >
> >    Based on what you sent yes it should take little time; if you send something I could run I could try to see why it would be slow.
> >
> >   Barry
> >
> > >
> > > Could you please have a look at the files attached.
> > >
> > >
> > >
> > >
> > >
> > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > >   You have two hundred processes banging on the same file on the same filesystem fighting with each other to read in the matrix. That's like hiring two hundred guys to dig a ditch but making them share one shovel; you won't get speed up, you'll get speed down.
> > >
> > >
> > >   Make a simple sequential PETSc program that reads in the matrix and then store it with MatView() in binary format. Then in your parallel program use MatLoad() to read the matrix in (it will take a few seconds to read in the matrix in that case).
> > >
> > >   Barry
> > >
> > >
> > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > > I've successfully diagonalized 100 million dimensional matrix in 2hours.
> > > >
> > > > But, the problem is when i try to diagonailze (similar) a smaller metrix of
> > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20 processors each (similar to the 100 million case), the matrix assembly itself
> > > > takes 2h !
> > > >
> > > > Could someone point out the mistakes that i'm making ?
> > > >
> > > > i attach the source code and the output with the mail.
> > > >
> > > > thanks,
> > > >  Vijay
> > > >
> > > >
> > > > #include <slepceps.h>
> > > > #include <petsctime.h>
> > > >
> > > > #undef __FUNCT__
> > > > #define __FUNCT__ "main"
> > > > int main(int argc,char **argv)
> > > > {
> > > >   Mat            A;           /* problem matrix */
> > > >   EPS            eps;         /* eigenproblem solver context */
> > > >   EPSType        type;
> > > >   PetscReal      error,tol,re,im;
> > > >   PetscScalar    kr,ki,value[52];
> > > >   Vec            xr,xi;
> > > >   PetscInt       n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > > >   PetscInt       d_nz,o_nz;
> > > >   PetscLogDouble t1,t2,tt1,tt2;
> > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > > >   PetscErrorCode ierr;
> > > >   PetscScalar    eigr;
> > > >   PetscScalar    eigi;
> > > >   PetscScalar    * data;
> > > >   Vec          Vr,Vi;
> > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > > >   PetscViewer    viewer;
> > > >   PetscBool      ishermitian;
> > > >   int            mpiid;
> > > >   long int            kk,iii;
> > > >   long int            tcountcol,tcol[52];
> > > >   float          val[52];
> > > >   long int            ntrou=1;
> > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >   double           xtt[52]   ={-1.0,0.,0.,
> > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > >
> > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem, n=%D\n\n",n);CHKERRQ(ierr);
> > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > > >
> > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > >   for (i=Istart; i<Iend; i++) {
> > > >       tcountcol=0;
> > > >       iii=i+1;
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >       }
> > > >       unit_l1_(
> > > >             l1,
> > > >             l2,
> > > >             ktyp,
> > > >             &iii,
> > > >             xjjxy,
> > > >             xjjz ,
> > > >             xtt  ,
> > > >             &tcountcol,
> > > >             &ntrou,
> > > >             tcol,
> > > >             val);
> > > >       for(kk=0;kk<52;kk++){
> > > >           value[kk] = val[kk];
> > > >           col[kk] = tcol[kk]-1;
> > > >       }
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > > >       }
> > > >       countcol=tcountcol;
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >       }
> > > >     ierr = MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > > >       if(i%5 == 0 && mpiid==0){
> > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > > >       }
> > > >   }
> > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > >
> > > >
> > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > > >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > > >   tol = 1.e-8;
> > > >   maxit = 10000000;
> > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > > >
> > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used: %f\n",t2-t1);CHKERRQ(ierr);
> > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method: %D\n",its);CHKERRQ(ierr);
> > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method: %s\n\n",type);CHKERRQ(ierr);
> > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > > >
> > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > > >   /*
> > > >   EPSGetConverged(eps,&nconv);
> > > >   if (nconv>0) {
> > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > > >     EPSIsHermitian(eps,&ishermitian);
> > > >     for (i=0;i<nconv;i++) {
> > > >       EPSGetEigenvector(eps,i,xr,xi);
> > > >       VecView(xr,viewer);
> > > > #if !defined(PETSC_USE_COMPLEX)
> > > >       if (!ishermitian) { VecView(xi,viewer); }
> > > > #endif
> > > >     }
> > > >     PetscViewerDestroy(&viewer);
> > > >   }
> > > >   */
> > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > > >   ierr = SlepcFinalize();
> > > >   return 0;
> > > > }
> > > >
> > > > OUTPUT:
> > > >  Time used to build the matrix: 1914.729022
> > > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3238253 unneeded,493497 used
> > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197999 unneeded,533751 used
> > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243809 unneeded,487941 used
> > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3243446 unneeded,488258 used
> > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3237359 unneeded,494391 used
> > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3245082 unneeded,486622 used
> > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3217795 unneeded,513955 used
> > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3208633 unneeded,523117 used
> > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3209020 unneeded,522684 used
> > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3216305 unneeded,515445 used
> > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3201643 unneeded,530107 used
> > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3234040 unneeded,497664 used
> > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243435 unneeded,488315 used
> > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3249858 unneeded,481846 used
> > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3207727 unneeded,524023 used
> > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3220647 unneeded,511103 used
> > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197939 unneeded,533811 used
> > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3179441 unneeded,552309 used
> > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3197890 unneeded,533814 used
> > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3200381 unneeded,531369 used
> > > >
> > > > <ex1.c><inp.log>
> > >
> > >
> >
> >
> > <1_Warning.txt>
> 
> 
> <ex1.c><makefile>


From thronesf at gmail.com  Thu May  7 15:39:20 2015
From: thronesf at gmail.com (Sharp Stone)
Date: Thu, 7 May 2015 16:39:20 -0400
Subject: [petsc-users] matrix elements out of range
Message-ID: <CABJpBAx34id_b=oSnvk0247iOFu5x=UMMRqmDc1-Y-XOTLri-Q@mail.gmail.com>

Hi all,

For the linear solver, if I use the standard 7-point differencing scheme
and use 2-point boundary conditions, it works fine. But when I change the
boundary condition, involving 3-point differencing form at the boundary,
such as x[i+2]-2*x[i+1]+x[i]=0, assuming i is boundary point number, I
always get the errors like
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Inserting a new nonzero into matrix

My boundary code is as follows.
if(i==0) {
 v[0] =  1.0;    col[0].i = i;
 v[1] = -2.0;    col[1].i = i+1;
 v[2] =  1.0;    col[2].i = i+2;
 MatSetValuesStencil(appctx->A,1,&row,3,col,v,INSERT_VALUES);
}

Does anyone have some suggestions? Thank you very much in advance.

-- 
Best regards,

Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/3562b32e/attachment.html>

From knepley at gmail.com  Thu May  7 15:40:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 7 May 2015 15:40:47 -0500
Subject: [petsc-users] matrix elements out of range
In-Reply-To: <CABJpBAx34id_b=oSnvk0247iOFu5x=UMMRqmDc1-Y-XOTLri-Q@mail.gmail.com>
References: <CABJpBAx34id_b=oSnvk0247iOFu5x=UMMRqmDc1-Y-XOTLri-Q@mail.gmail.com>
Message-ID: <CAMYG4GkqhUwwQjrMYAaye=cNz8T4mVX9mm5JuYrxTGkpgS+Cmw@mail.gmail.com>

On Thu, May 7, 2015 at 3:39 PM, Sharp Stone <thronesf at gmail.com> wrote:

> Hi all,
>
> For the linear solver, if I use the standard 7-point differencing scheme
> and use 2-point boundary conditions, it works fine. But when I change the
> boundary condition, involving 3-point differencing form at the boundary,
> such as x[i+2]-2*x[i+1]+x[i]=0, assuming i is boundary point number, I
> always get the errors like
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero into matrix
>

When you create the DMDA, you have to tell it that the stencil has width 2.

   Matt


> My boundary code is as follows.
> if(i==0) {
>  v[0] =  1.0;    col[0].i = i;
>  v[1] = -2.0;    col[1].i = i+1;
>  v[2] =  1.0;    col[2].i = i+2;
>  MatSetValuesStencil(appctx->A,1,&row,3,col,v,INSERT_VALUES);
> }
>
> Does anyone have some suggestions? Thank you very much in advance.
>
> --
> Best regards,
>
> Feng
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/0ad63872/attachment.html>

From jychang48 at gmail.com  Thu May  7 22:31:31 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Thu, 7 May 2015 22:31:31 -0500
Subject: [petsc-users] Structured grid in DMPlex?
Message-ID: <CAP2=TMiBe6FqFCKdyvHi6P6LNWL_OSt3EuDAsWy+-ab3KQmfNw@mail.gmail.com>

Hello everyone,

If I know for a fact that I will be using structured grids AND still want
the DMPlex data structure, what's the best/fastest way of achieving this?
Should I:

a) Invoke DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE,
DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,-3,-3,PETSC_DECIDE,PETSC_DECIDE,1,1,NULL,NULL,&da);
to get the parallel cell-vertex layout and (somehow) translate this into a
DMPlex?

b) Manually set the local DMPlex chart/DAG and assign the SF along the
appropriate processors?

c) Invoke some DMDA/DMPlex function or set of functions that already exist
for this purpose?

d) Something else?

I know I could just input or create a coarse DAG and invoke -dm_refine, but
I was wondering if there is an alternative to this method.

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150507/b546ccba/attachment.html>

From a.croucher at auckland.ac.nz  Fri May  8 01:24:01 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Fri, 08 May 2015 18:24:01 +1200
Subject: [petsc-users] DMPlex and HDF5 vector ordering
Message-ID: <554C5681.7020704@auckland.ac.nz>

hi,

I create a Vec on a DMPlex using DMPlexCreateGlobalVec(), then write it 
to HDF5 using PetscViewerHDF5Open() and VecView().

I then try to read it back in later (in another program, but using the 
same DMPlex) using PetscViewerHDF5Open() and VecLoad().

It looks like the ordering of the final vector entries in the second 
program depends on how many processors I use. If they are the same in 
both programs, I get the right ordering, but if they aren't, I don't. Is 
that expected? If so, is there any way to guarantee the right ordering 
when I read the Vec back in?

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611


From jychang48 at gmail.com  Fri May  8 01:48:52 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 8 May 2015 01:48:52 -0500
Subject: [petsc-users]  DMPlex and HDF5 vector ordering
In-Reply-To: <554C5681.7020704@auckland.ac.nz>
References: <554C5681.7020704@auckland.ac.nz>
Message-ID: <CAP2=TMitVLADVp9=j5emnHx2AVegi24MG2JXGgf92Jys=C1adQ@mail.gmail.com>

I also had the same issue. My current work around is the following.

1) Run the first DMPlex program on one process and write the vector into
HDF5.

2) Run the second DMPlex program with any number of processes but do the
following:

3) After you create the initial DMPlex on rank 0, but before distributing
it, duplicate it and create its petscsection and vector.

4) Load the HDF5 file into that vector. At this point the ordering is the
same.

5) Distribute the original DM and save the PetscSF.

6) Call DMPlexDistributeField() to distribute the vector.


This will guarantee the right ordering for the second program no matter how
many processes it uses. Only drawback is that the first program has to be
run in serial. I am also looking for a better way. Matt any thoughts?

Thanks,
Justin

On Friday, May 8, 2015, Adrian Croucher <a.croucher at auckland.ac.nz
<javascript:_e(%7B%7D,'cvml','a.croucher at auckland.ac.nz');>> wrote:

> hi,
>
> I create a Vec on a DMPlex using DMPlexCreateGlobalVec(), then write it to
> HDF5 using PetscViewerHDF5Open() and VecView().
>
> I then try to read it back in later (in another program, but using the
> same DMPlex) using PetscViewerHDF5Open() and VecLoad().
>
> It looks like the ordering of the final vector entries in the second
> program depends on how many processors I use. If they are the same in both
> programs, I get the right ordering, but if they aren't, I don't. Is that
> expected? If so, is there any way to guarantee the right ordering when I
> read the Vec back in?
>
> - Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/8ba0f106/attachment.html>

From vijay.gopal.c at gmail.com  Fri May  8 03:27:54 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Fri, 8 May 2015 09:27:54 +0100
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
	<8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
Message-ID: <CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>

Thanks for testing.

My problem is not with MatSetValues(), it is
MatAssemblyBegin()/MatAssemblyEnd() that takes forever to finish.
MatSetValues() is quite fast as you noticed.

Actually, unit_l1_ is the subroutine that generates the nonzero terms (per
row) of my hamiltonian. For large matrices this should be time consuming if
done on a single processor.
This is the reason that i use a lot of processors to divide the work of
calling unit_l1_ multiple times.
For example, if I use 171 processors over 10 nodes, unit_l1_ finishes in
about 40 min. It is after this step at the MatAssemblyBegin() and
MatAssemblyEnd() that the
PETSc code get's stuck.

If it's not too much trouble, could you perhaps test with more processors ?

thanks again.

On Thu, May 7, 2015 at 9:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   I built the code with gfortran and optimization on my mac and run it on
> one process. It is spending all its time in the unit_l1_ routine and almost
> no time in the MatSetValues() in fact if I comment out the MatSetValues()
> line it runs just as slow as before. How can you know the time is spend in
> the MatSetValues()? Have you timed it also with the MatSetValues() time
> commented out?
>
>    Barry
>
> > On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> >
> > You are right ! I am using the IRPF90 tool to write my program in
> FORTRAN.
> >
> > To use IRPF90 you can clone the github repository and get the irpf90
> binary from the bin directory.
> > The code to generate the library that contains UNIT_L1_ can be found at:
> https://github.com/v1j4y/slepc_version.git
> > just clone this repository and do:
> >
> > make
> > make irpf90.a
> >
> > That's it !
> >
> > I'm resending you the PETSc files along with the makefile. you'll need
> the irpf90.a library to make the ex1 executable.
> >
> > Hopefully this time it'll work.
> > Please let me know if something goes wrong.
> >
> > On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Our stupid mail system blocked the attachment because it contained
> irpf90.a   Is this a library generated with
> https://github.com/scemama/irpf90 ?
> > You need to email everything that goes into making that library because
> it is unlikely your .a library will work on my machine. What is this thing
> and can you just tar up the directory that builds it and I can generate
> irpf90.a
> >
> >   Barry
> >
> > > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > >
> > > Ok so here goes.
> > >
> > > I've attached a tarball with a directory which contains the required
> files to compile ex1.c
> > > The UNIT_L1_ subroutine generates the nonzero elements of my
> hamiltonian given a row, it is contained in the
> > > irpf90.a library. The actual fortran code is quite ugly.
> > > Please tell me if you are able to compile and run the codes.
> > >
> > > Thanks a lot,
> > >  Vijay
> > >
> > > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > > >
> > > > Please let me clarify,
> > > >
> > > > @Matt:
> > > > I've followed the proper manner to rapid assembly.
> > > > That is :
> > > >   1. preallocation
> > > >   2. nonzero ownership
> > > > Despite of having done the proper preallocation (as you can see from
> the output file),
> > > > the matrix takes ages to assemble, whereas the same code works for a
> larger matrix with (100,000,000) elements taking 2 min to assemble !
> > > >
> > > > @Barry:
> > > > I am not touching any file in the code that I attachd.
> > > > What i'm doing is that each processor will (according to it's
> ownership range) get it's nonzero entries
> > > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for
> their respective rows.
> > > > Again to be clear I"m NOT doing file I/O anywhere.
> > >
> > >   Oh sorry, where is the fortran code? I assumed the fortran code was
> reading from a file.
> > >
> > >    Based on what you sent yes it should take little time; if you send
> something I could run I could try to see why it would be slow.
> > >
> > >   Barry
> > >
> > > >
> > > > Could you please have a look at the files attached.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > >   You have two hundred processes banging on the same file on the
> same filesystem fighting with each other to read in the matrix. That's like
> hiring two hundred guys to dig a ditch but making them share one shovel;
> you won't get speed up, you'll get speed down.
> > > >
> > > >
> > > >   Make a simple sequential PETSc program that reads in the matrix
> and then store it with MatView() in binary format. Then in your parallel
> program use MatLoad() to read the matrix in (it will take a few seconds to
> read in the matrix in that case).
> > > >
> > > >   Barry
> > > >
> > > >
> > > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > > > I've successfully diagonalized 100 million dimensional matrix in
> 2hours.
> > > > >
> > > > > But, the problem is when i try to diagonailze (similar) a smaller
> metrix of
> > > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
> running 20 processors each (similar to the 100 million case), the matrix
> assembly itself
> > > > > takes 2h !
> > > > >
> > > > > Could someone point out the mistakes that i'm making ?
> > > > >
> > > > > i attach the source code and the output with the mail.
> > > > >
> > > > > thanks,
> > > > >  Vijay
> > > > >
> > > > >
> > > > > #include <slepceps.h>
> > > > > #include <petsctime.h>
> > > > >
> > > > > #undef __FUNCT__
> > > > > #define __FUNCT__ "main"
> > > > > int main(int argc,char **argv)
> > > > > {
> > > > >   Mat            A;           /* problem matrix */
> > > > >   EPS            eps;         /* eigenproblem solver context */
> > > > >   EPSType        type;
> > > > >   PetscReal      error,tol,re,im;
> > > > >   PetscScalar    kr,ki,value[52];
> > > > >   Vec            xr,xi;
> > > > >   PetscInt
>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > > > >   PetscInt       d_nz,o_nz;
> > > > >   PetscLogDouble t1,t2,tt1,tt2;
> > > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > > > >   PetscErrorCode ierr;
> > > > >   PetscScalar    eigr;
> > > > >   PetscScalar    eigi;
> > > > >   PetscScalar    * data;
> > > > >   Vec          Vr,Vi;
> > > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > > > >   PetscViewer    viewer;
> > > > >   PetscBool      ishermitian;
> > > > >   int            mpiid;
> > > > >   long int            kk,iii;
> > > > >   long int            tcountcol,tcol[52];
> > > > >   float          val[52];
> > > > >   long int            ntrou=1;
> > > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
> 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
> 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20,
> 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2,
> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >   double           xtt[52]   ={-1.0,0.,0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >
> > > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
> n=%D\n\n",n);CHKERRQ(ierr);
> > > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > > > //ierr =
> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > > > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > > > >   ierr =
> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > > >   for (i=Istart; i<Iend; i++) {
> > > > >       tcountcol=0;
> > > > >       iii=i+1;
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >       }
> > > > >       unit_l1_(
> > > > >             l1,
> > > > >             l2,
> > > > >             ktyp,
> > > > >             &iii,
> > > > >             xjjxy,
> > > > >             xjjz ,
> > > > >             xtt  ,
> > > > >             &tcountcol,
> > > > >             &ntrou,
> > > > >             tcol,
> > > > >             val);
> > > > >       for(kk=0;kk<52;kk++){
> > > > >           value[kk] = val[kk];
> > > > >           col[kk] = tcol[kk]-1;
> > > > >       }
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid:
> %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > > > >       }
> > > > >       countcol=tcountcol;
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >       }
> > > > >     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > > > >       }
> > > > >   }
> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > > >
> > > > >
> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the
> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > > > >   ierr =
> EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > > > >   tol = 1.e-8;
> > > > >   maxit = 10000000;
> > > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
> %f\n",t2-t1);CHKERRQ(ierr);
> > > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of
> the method: %D\n",its);CHKERRQ(ierr);
> > > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
> %s\n\n",type);CHKERRQ(ierr);
> > > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
> eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition:
> tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > > > >   /*
> > > > >   EPSGetConverged(eps,&nconv);
> > > > >   if (nconv>0) {
> > > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > > > >     EPSIsHermitian(eps,&ishermitian);
> > > > >     for (i=0;i<nconv;i++) {
> > > > >       EPSGetEigenvector(eps,i,xr,xi);
> > > > >       VecView(xr,viewer);
> > > > > #if !defined(PETSC_USE_COMPLEX)
> > > > >       if (!ishermitian) { VecView(xi,viewer); }
> > > > > #endif
> > > > >     }
> > > > >     PetscViewerDestroy(&viewer);
> > > > >   }
> > > > >   */
> > > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > > > >   ierr = SlepcFinalize();
> > > > >   return 0;
> > > > > }
> > > > >
> > > > > OUTPUT:
> > > > >  Time used to build the matrix: 1914.729022
> > > > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
> mallocs.
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3238253 unneeded,493497 used
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197999 unneeded,533751 used
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243809 unneeded,487941 used
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3243446 unneeded,488258 used
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3237359 unneeded,494391 used
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3245082 unneeded,486622 used
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3217795 unneeded,513955 used
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3208633 unneeded,523117 used
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3209020 unneeded,522684 used
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3216305 unneeded,515445 used
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3201643 unneeded,530107 used
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3234040 unneeded,497664 used
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3243435 unneeded,488315 used
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3249858 unneeded,481846 used
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3207727 unneeded,524023 used
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3220647 unneeded,511103 used
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3197939 unneeded,533811 used
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3179441 unneeded,552309 used
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage
> space: 3197890 unneeded,533814 used
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
> MatSetValues() is 0
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
> space: 3200381 unneeded,531369 used
> > > > >
> > > > > <ex1.c><inp.log>
> > > >
> > > >
> > >
> > >
> > > <1_Warning.txt>
> >
> >
> > <ex1.c><makefile>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/4266b915/attachment-0001.html>

From knepley at gmail.com  Fri May  8 06:21:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 06:21:35 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
	<8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
	<CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
Message-ID: <CAMYG4GktzqB3-rWiOAcZjiAmqswqyeeXuK5SqF1rK6wjw_Hg-A@mail.gmail.com>

On Fri, May 8, 2015 at 3:27 AM, Vijay Gopal Chilkuri <
vijay.gopal.c at gmail.com> wrote:

> Thanks for testing.
>
> My problem is not with MatSetValues(), it is
> MatAssemblyBegin()/MatAssemblyEnd() that takes forever to finish.
> MatSetValues() is quite fast as you noticed.
>
> Actually, unit_l1_ is the subroutine that generates the nonzero terms (per
> row) of my hamiltonian. For large matrices this should be time consuming if
> done on a single processor.
> This is the reason that i use a lot of processors to divide the work of
> calling unit_l1_ multiple times.
> For example, if I use 171 processors over 10 nodes, unit_l1_ finishes in
> about 40 min. It is after this step at the MatAssemblyBegin() and
> MatAssemblyEnd() that the
> PETSc code get's stuck.
>

Are you generating values on a random process, or on the process that owns
the row? If you
generate them anywhere, then MatAssemblyEnd() can have a large amount of
data to
communicate.

  Matt


> If it's not too much trouble, could you perhaps test with more processors ?
>
> thanks again.
>
> On Thu, May 7, 2015 at 9:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   I built the code with gfortran and optimization on my mac and run it on
>> one process. It is spending all its time in the unit_l1_ routine and almost
>> no time in the MatSetValues() in fact if I comment out the MatSetValues()
>> line it runs just as slow as before. How can you know the time is spend in
>> the MatSetValues()? Have you timed it also with the MatSetValues() time
>> commented out?
>>
>>    Barry
>>
>> > On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <
>> vijay.gopal.c at gmail.com> wrote:
>> >
>> > You are right ! I am using the IRPF90 tool to write my program in
>> FORTRAN.
>> >
>> > To use IRPF90 you can clone the github repository and get the irpf90
>> binary from the bin directory.
>> > The code to generate the library that contains UNIT_L1_ can be found
>> at: https://github.com/v1j4y/slepc_version.git
>> > just clone this repository and do:
>> >
>> > make
>> > make irpf90.a
>> >
>> > That's it !
>> >
>> > I'm resending you the PETSc files along with the makefile. you'll need
>> the irpf90.a library to make the ex1 executable.
>> >
>> > Hopefully this time it'll work.
>> > Please let me know if something goes wrong.
>> >
>> > On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> >   Our stupid mail system blocked the attachment because it contained
>> irpf90.a   Is this a library generated with
>> https://github.com/scemama/irpf90 ?
>> > You need to email everything that goes into making that library because
>> it is unlikely your .a library will work on my machine. What is this thing
>> and can you just tar up the directory that builds it and I can generate
>> irpf90.a
>> >
>> >   Barry
>> >
>> > > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <
>> vijay.gopal.c at gmail.com> wrote:
>> > >
>> > > Ok so here goes.
>> > >
>> > > I've attached a tarball with a directory which contains the required
>> files to compile ex1.c
>> > > The UNIT_L1_ subroutine generates the nonzero elements of my
>> hamiltonian given a row, it is contained in the
>> > > irpf90.a library. The actual fortran code is quite ugly.
>> > > Please tell me if you are able to compile and run the codes.
>> > >
>> > > Thanks a lot,
>> > >  Vijay
>> > >
>> > > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > >
>> > > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
>> vijay.gopal.c at gmail.com> wrote:
>> > > >
>> > > > Please let me clarify,
>> > > >
>> > > > @Matt:
>> > > > I've followed the proper manner to rapid assembly.
>> > > > That is :
>> > > >   1. preallocation
>> > > >   2. nonzero ownership
>> > > > Despite of having done the proper preallocation (as you can see
>> from the output file),
>> > > > the matrix takes ages to assemble, whereas the same code works for
>> a larger matrix with (100,000,000) elements taking 2 min to assemble !
>> > > >
>> > > > @Barry:
>> > > > I am not touching any file in the code that I attachd.
>> > > > What i'm doing is that each processor will (according to it's
>> ownership range) get it's nonzero entries
>> > > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for
>> their respective rows.
>> > > > Again to be clear I"m NOT doing file I/O anywhere.
>> > >
>> > >   Oh sorry, where is the fortran code? I assumed the fortran code was
>> reading from a file.
>> > >
>> > >    Based on what you sent yes it should take little time; if you send
>> something I could run I could try to see why it would be slow.
>> > >
>> > >   Barry
>> > >
>> > > >
>> > > > Could you please have a look at the files attached.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > > >
>> > > >   You have two hundred processes banging on the same file on the
>> same filesystem fighting with each other to read in the matrix. That's like
>> hiring two hundred guys to dig a ditch but making them share one shovel;
>> you won't get speed up, you'll get speed down.
>> > > >
>> > > >
>> > > >   Make a simple sequential PETSc program that reads in the matrix
>> and then store it with MatView() in binary format. Then in your parallel
>> program use MatLoad() to read the matrix in (it will take a few seconds to
>> read in the matrix in that case).
>> > > >
>> > > >   Barry
>> > > >
>> > > >
>> > > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
>> vijay.gopal.c at gmail.com> wrote:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
>> > > > > I've successfully diagonalized 100 million dimensional matrix in
>> 2hours.
>> > > > >
>> > > > > But, the problem is when i try to diagonailze (similar) a smaller
>> metrix of
>> > > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
>> running 20 processors each (similar to the 100 million case), the matrix
>> assembly itself
>> > > > > takes 2h !
>> > > > >
>> > > > > Could someone point out the mistakes that i'm making ?
>> > > > >
>> > > > > i attach the source code and the output with the mail.
>> > > > >
>> > > > > thanks,
>> > > > >  Vijay
>> > > > >
>> > > > >
>> > > > > #include <slepceps.h>
>> > > > > #include <petsctime.h>
>> > > > >
>> > > > > #undef __FUNCT__
>> > > > > #define __FUNCT__ "main"
>> > > > > int main(int argc,char **argv)
>> > > > > {
>> > > > >   Mat            A;           /* problem matrix */
>> > > > >   EPS            eps;         /* eigenproblem solver context */
>> > > > >   EPSType        type;
>> > > > >   PetscReal      error,tol,re,im;
>> > > > >   PetscScalar    kr,ki,value[52];
>> > > > >   Vec            xr,xi;
>> > > > >   PetscInt
>>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
>> > > > >   PetscInt       d_nz,o_nz;
>> > > > >   PetscLogDouble t1,t2,tt1,tt2;
>> > > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
>> > > > >   PetscErrorCode ierr;
>> > > > >   PetscScalar    eigr;
>> > > > >   PetscScalar    eigi;
>> > > > >   PetscScalar    * data;
>> > > > >   Vec          Vr,Vi;
>> > > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
>> > > > >   PetscViewer    viewer;
>> > > > >   PetscBool      ishermitian;
>> > > > >   int            mpiid;
>> > > > >   long int            kk,iii;
>> > > > >   long int            tcountcol,tcol[52];
>> > > > >   float          val[52];
>> > > > >   long int            ntrou=1;
>> > > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
>> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
>> 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>> > > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
>> 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20,
>> 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>> > > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2,
>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
>> > > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>> > > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>> > > > >   double           xtt[52]   ={-1.0,0.,0.,
>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>> > > > >
>> > > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
>> n=%D\n\n",n);CHKERRQ(ierr);
>> > > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>> > > > > //ierr =
>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
>> > > > >   ierr =
>> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>> > > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>> > > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
>> > > > >
>> > > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>> > > > >   ierr =
>> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>> > > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>> > > > >   for (i=Istart; i<Iend; i++) {
>> > > > >       tcountcol=0;
>> > > > >       iii=i+1;
>> > > > >       if(i%5 == 0 && mpiid==0){
>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>> > > > >       }
>> > > > >       unit_l1_(
>> > > > >             l1,
>> > > > >             l2,
>> > > > >             ktyp,
>> > > > >             &iii,
>> > > > >             xjjxy,
>> > > > >             xjjz ,
>> > > > >             xtt  ,
>> > > > >             &tcountcol,
>> > > > >             &ntrou,
>> > > > >             tcol,
>> > > > >             val);
>> > > > >       for(kk=0;kk<52;kk++){
>> > > > >           value[kk] = val[kk];
>> > > > >           col[kk] = tcol[kk]-1;
>> > > > >       }
>> > > > >       if(i%5 == 0 && mpiid==0){
>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid:
>> %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
>> > > > >       }
>> > > > >       countcol=tcountcol;
>> > > > >       if(i%5 == 0 && mpiid==0){
>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>> > > > >       }
>> > > > >     ierr =
>> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>> > > > >       if(i%5 == 0 && mpiid==0){
>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid:
>> %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
>> > > > >       }
>> > > > >   }
>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the
>> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>> > > > >
>> > > > >
>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>> > > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> > > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the
>> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>> > > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
>> > > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
>> > > > >
>> > > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
>> > > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
>> > > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
>> > > > >   ierr =
>> EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
>> > > > >
>> > > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
>> > > > >   tol = 1.e-8;
>> > > > >   maxit = 10000000;
>> > > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
>> > > > >
>> > > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
>> > > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
>> > > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
>> %f\n",t2-t1);CHKERRQ(ierr);
>> > > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of
>> the method: %D\n",its);CHKERRQ(ierr);
>> > > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
>> %s\n\n",type);CHKERRQ(ierr);
>> > > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
>> eigenvalues: %D\n",nev);CHKERRQ(ierr);
>> > > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition:
>> tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
>> > > > >
>> > > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
>> > > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
>> > > > >   /*
>> > > > >   EPSGetConverged(eps,&nconv);
>> > > > >   if (nconv>0) {
>> > > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
>> > > > >     EPSIsHermitian(eps,&ishermitian);
>> > > > >     for (i=0;i<nconv;i++) {
>> > > > >       EPSGetEigenvector(eps,i,xr,xi);
>> > > > >       VecView(xr,viewer);
>> > > > > #if !defined(PETSC_USE_COMPLEX)
>> > > > >       if (!ishermitian) { VecView(xi,viewer); }
>> > > > > #endif
>> > > > >     }
>> > > > >     PetscViewerDestroy(&viewer);
>> > > > >   }
>> > > > >   */
>> > > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
>> > > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
>> > > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
>> > > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
>> > > > >   ierr = SlepcFinalize();
>> > > > >   return 0;
>> > > > > }
>> > > > >
>> > > > > OUTPUT:
>> > > > >  Time used to build the matrix: 1914.729022
>> > > > > [0] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [6] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [182] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [100] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [60] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [140] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [120] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [162] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [20] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [80] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [40] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [7] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [185] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [101] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [61] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [142] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [130] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [166] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [21] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [81] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [44] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [9] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [187] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [105] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [62] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [143] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [132] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [169] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [24] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [84] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [45] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [10] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [188] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [109] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [63] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [148] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [133] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [170] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [25] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [86] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [47] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [11] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [190] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [111] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [64] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [156] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [134] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [171] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [26] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [91] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [48] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [12] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [192] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [112] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [65] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [157] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [135] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [174] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [28] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [13] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [194] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [113] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [66] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [158] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [136] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [160] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [29] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [92] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [53] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [14] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [195] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [114] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [67] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [141] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [137] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [161] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [32] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [93] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [54] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [15] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [196] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [115] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [68] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [144] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [138] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [163] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [33] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [94] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [55] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [17] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [198] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [116] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [69] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [145] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [139] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [164] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [34] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [95] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [56] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [1] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [199] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [119] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [70] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [146] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [121] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [165] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [35] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [99] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [59] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [2] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [180] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [102] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [71] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [147] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [122] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [167] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [36] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [82] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [41] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [3] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [181] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [103] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [72] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [149] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [123] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [168] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [37] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [83] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [42] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [4] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [183] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [104] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [73] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [150] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [124] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [172] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [38] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [85] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [43] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [5] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [184] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [106] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [74] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [151] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [125] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [173] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [39] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [87] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [46] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [8] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [186] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [107] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [75] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [152] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [126] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [175] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [22] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [88] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [49] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [16] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [189] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [108] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [76] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [153] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [127] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [176] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [23] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [89] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [50] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [18] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [191] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [110] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [77] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [154] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [128] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [177] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [27] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [90] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [51] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [19] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [193] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [117] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [78] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [155] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [129] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [178] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [30] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [96] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [52] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [197] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [118] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [79] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [159] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [131] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [179] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [31] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [97] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [57] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [98] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [58] MatStashScatterBegin_Private(): No of messages: 0
>> > > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>> mallocs.
>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>> storage space: 3238253 unneeded,493497 used
>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3197999 unneeded,533751 used
>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>> storage space: 3243809 unneeded,487941 used
>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3243446 unneeded,488258 used
>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>> > > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>> storage space: 3237359 unneeded,494391 used
>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3245082 unneeded,486622 used
>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3217795 unneeded,513955 used
>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3208633 unneeded,523117 used
>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3209020 unneeded,522684 used
>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>> storage space: 3216305 unneeded,515445 used
>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3201643 unneeded,530107 used
>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3234040 unneeded,497664 used
>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>> > > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>> storage space: 3243435 unneeded,488315 used
>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3249858 unneeded,481846 used
>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3207727 unneeded,524023 used
>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3220647 unneeded,511103 used
>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>> > > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3197939 unneeded,533811 used
>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3179441 unneeded,552309 used
>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>> storage space: 3197890 unneeded,533814 used
>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>> MatSetValues() is 0
>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>> > > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>> > > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>> > > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>> space: 3200381 unneeded,531369 used
>> > > > >
>> > > > > <ex1.c><inp.log>
>> > > >
>> > > >
>> > >
>> > >
>> > > <1_Warning.txt>
>> >
>> >
>> > <ex1.c><makefile>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/0e857965/attachment-0001.html>

From vijay.gopal.c at gmail.com  Fri May  8 07:20:12 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Fri, 8 May 2015 13:20:12 +0100
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAMYG4GktzqB3-rWiOAcZjiAmqswqyeeXuK5SqF1rK6wjw_Hg-A@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
	<8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
	<CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
	<CAMYG4GktzqB3-rWiOAcZjiAmqswqyeeXuK5SqF1rK6wjw_Hg-A@mail.gmail.com>
Message-ID: <CAJa9Fi9siu+AB1CsYm2TVwXymxbtRbpRSJZOM30bLrx9hpZLkg@mail.gmail.com>

Hi matt,

I'm not generating values on random processors.
here is the relevant part of the code:


>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>   for (i=Istart; i<Iend; i++) {
>       tcountcol=0;
>       iii=i+1;
>       unit_l1_(
>             l1,
>             l2,
>             ktyp,
>             &iii,
>             xjjxy,
>             xjjz ,
>             xtt  ,
>             &tcountcol,
>             &ntrou,
>             tcol,
>             val);
>       for(kk=0;kk<52;kk++){
>           value[kk] = val[kk];
>           col[kk] = tcol[kk]-1;
>       }
>       countcol=tcountcol;
>     ierr =
> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>   }


As can be seen, I get the relevant rows attributed to each processor and
only MatSetValues() for the rows owned by the corresponding processors.
That's why it is surprising that MatAssemnly takes a long time !


On Fri, May 8, 2015 at 12:21 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 8, 2015 at 3:27 AM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
>
>> Thanks for testing.
>>
>> My problem is not with MatSetValues(), it is
>> MatAssemblyBegin()/MatAssemblyEnd() that takes forever to finish.
>> MatSetValues() is quite fast as you noticed.
>>
>> Actually, unit_l1_ is the subroutine that generates the nonzero terms
>> (per row) of my hamiltonian. For large matrices this should be time
>> consuming if done on a single processor.
>> This is the reason that i use a lot of processors to divide the work of
>> calling unit_l1_ multiple times.
>> For example, if I use 171 processors over 10 nodes, unit_l1_ finishes in
>> about 40 min. It is after this step at the MatAssemblyBegin() and
>> MatAssemblyEnd() that the
>> PETSc code get's stuck.
>>
>
> Are you generating values on a random process, or on the process that owns
> the row? If you
> generate them anywhere, then MatAssemblyEnd() can have a large amount of
> data to
> communicate.
>
>   Matt
>
>
>> If it's not too much trouble, could you perhaps test with more processors
>> ?
>>
>> thanks again.
>>
>> On Thu, May 7, 2015 at 9:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>   I built the code with gfortran and optimization on my mac and run it
>>> on one process. It is spending all its time in the unit_l1_ routine and
>>> almost no time in the MatSetValues() in fact if I comment out the
>>> MatSetValues() line it runs just as slow as before. How can you know the
>>> time is spend in the MatSetValues()? Have you timed it also with the
>>> MatSetValues() time commented out?
>>>
>>>    Barry
>>>
>>> > On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <
>>> vijay.gopal.c at gmail.com> wrote:
>>> >
>>> > You are right ! I am using the IRPF90 tool to write my program in
>>> FORTRAN.
>>> >
>>> > To use IRPF90 you can clone the github repository and get the irpf90
>>> binary from the bin directory.
>>> > The code to generate the library that contains UNIT_L1_ can be found
>>> at: https://github.com/v1j4y/slepc_version.git
>>> > just clone this repository and do:
>>> >
>>> > make
>>> > make irpf90.a
>>> >
>>> > That's it !
>>> >
>>> > I'm resending you the PETSc files along with the makefile. you'll need
>>> the irpf90.a library to make the ex1 executable.
>>> >
>>> > Hopefully this time it'll work.
>>> > Please let me know if something goes wrong.
>>> >
>>> > On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >   Our stupid mail system blocked the attachment because it contained
>>> irpf90.a   Is this a library generated with
>>> https://github.com/scemama/irpf90 ?
>>> > You need to email everything that goes into making that library
>>> because it is unlikely your .a library will work on my machine. What is
>>> this thing and can you just tar up the directory that builds it and I can
>>> generate irpf90.a
>>> >
>>> >   Barry
>>> >
>>> > > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <
>>> vijay.gopal.c at gmail.com> wrote:
>>> > >
>>> > > Ok so here goes.
>>> > >
>>> > > I've attached a tarball with a directory which contains the required
>>> files to compile ex1.c
>>> > > The UNIT_L1_ subroutine generates the nonzero elements of my
>>> hamiltonian given a row, it is contained in the
>>> > > irpf90.a library. The actual fortran code is quite ugly.
>>> > > Please tell me if you are able to compile and run the codes.
>>> > >
>>> > > Thanks a lot,
>>> > >  Vijay
>>> > >
>>> > > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > >
>>> > > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
>>> vijay.gopal.c at gmail.com> wrote:
>>> > > >
>>> > > > Please let me clarify,
>>> > > >
>>> > > > @Matt:
>>> > > > I've followed the proper manner to rapid assembly.
>>> > > > That is :
>>> > > >   1. preallocation
>>> > > >   2. nonzero ownership
>>> > > > Despite of having done the proper preallocation (as you can see
>>> from the output file),
>>> > > > the matrix takes ages to assemble, whereas the same code works for
>>> a larger matrix with (100,000,000) elements taking 2 min to assemble !
>>> > > >
>>> > > > @Barry:
>>> > > > I am not touching any file in the code that I attachd.
>>> > > > What i'm doing is that each processor will (according to it's
>>> ownership range) get it's nonzero entries
>>> > > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for
>>> their respective rows.
>>> > > > Again to be clear I"m NOT doing file I/O anywhere.
>>> > >
>>> > >   Oh sorry, where is the fortran code? I assumed the fortran code
>>> was reading from a file.
>>> > >
>>> > >    Based on what you sent yes it should take little time; if you
>>> send something I could run I could try to see why it would be slow.
>>> > >
>>> > >   Barry
>>> > >
>>> > > >
>>> > > > Could you please have a look at the files attached.
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > > >
>>> > > >   You have two hundred processes banging on the same file on the
>>> same filesystem fighting with each other to read in the matrix. That's like
>>> hiring two hundred guys to dig a ditch but making them share one shovel;
>>> you won't get speed up, you'll get speed down.
>>> > > >
>>> > > >
>>> > > >   Make a simple sequential PETSc program that reads in the matrix
>>> and then store it with MatView() in binary format. Then in your parallel
>>> program use MatLoad() to read the matrix in (it will take a few seconds to
>>> read in the matrix in that case).
>>> > > >
>>> > > >   Barry
>>> > > >
>>> > > >
>>> > > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
>>> vijay.gopal.c at gmail.com> wrote:
>>> > > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
>>> > > > > I've successfully diagonalized 100 million dimensional matrix in
>>> 2hours.
>>> > > > >
>>> > > > > But, the problem is when i try to diagonailze (similar) a
>>> smaller metrix of
>>> > > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
>>> running 20 processors each (similar to the 100 million case), the matrix
>>> assembly itself
>>> > > > > takes 2h !
>>> > > > >
>>> > > > > Could someone point out the mistakes that i'm making ?
>>> > > > >
>>> > > > > i attach the source code and the output with the mail.
>>> > > > >
>>> > > > > thanks,
>>> > > > >  Vijay
>>> > > > >
>>> > > > >
>>> > > > > #include <slepceps.h>
>>> > > > > #include <petsctime.h>
>>> > > > >
>>> > > > > #undef __FUNCT__
>>> > > > > #define __FUNCT__ "main"
>>> > > > > int main(int argc,char **argv)
>>> > > > > {
>>> > > > >   Mat            A;           /* problem matrix */
>>> > > > >   EPS            eps;         /* eigenproblem solver context */
>>> > > > >   EPSType        type;
>>> > > > >   PetscReal      error,tol,re,im;
>>> > > > >   PetscScalar    kr,ki,value[52];
>>> > > > >   Vec            xr,xi;
>>> > > > >   PetscInt
>>>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
>>> > > > >   PetscInt       d_nz,o_nz;
>>> > > > >   PetscLogDouble t1,t2,tt1,tt2;
>>> > > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
>>> > > > >   PetscErrorCode ierr;
>>> > > > >   PetscScalar    eigr;
>>> > > > >   PetscScalar    eigi;
>>> > > > >   PetscScalar    * data;
>>> > > > >   Vec          Vr,Vi;
>>> > > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
>>> > > > >   PetscViewer    viewer;
>>> > > > >   PetscBool      ishermitian;
>>> > > > >   int            mpiid;
>>> > > > >   long int            kk,iii;
>>> > > > >   long int            tcountcol,tcol[52];
>>> > > > >   float          val[52];
>>> > > > >   long int            ntrou=1;
>>> > > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
>>> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
>>> 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>>> > > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
>>> 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19,
>>> 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
>>>  };
>>> > > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
>>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
>>> > > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>> > > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>> > > > >   double           xtt[52]   ={-1.0,0.,0.,
>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>> > > > >
>>> > > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
>>> n=%D\n\n",n);CHKERRQ(ierr);
>>> > > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>>> > > > > //ierr =
>>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
>>> > > > >   ierr =
>>> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>>> > > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>>> > > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
>>> > > > >
>>> > > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>>> > > > >   ierr =
>>> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>>> > > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>>> > > > >   for (i=Istart; i<Iend; i++) {
>>> > > > >       tcountcol=0;
>>> > > > >       iii=i+1;
>>> > > > >       if(i%5 == 0 && mpiid==0){
>>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>>> > > > >       }
>>> > > > >       unit_l1_(
>>> > > > >             l1,
>>> > > > >             l2,
>>> > > > >             ktyp,
>>> > > > >             &iii,
>>> > > > >             xjjxy,
>>> > > > >             xjjz ,
>>> > > > >             xtt  ,
>>> > > > >             &tcountcol,
>>> > > > >             &ntrou,
>>> > > > >             tcol,
>>> > > > >             val);
>>> > > > >       for(kk=0;kk<52;kk++){
>>> > > > >           value[kk] = val[kk];
>>> > > > >           col[kk] = tcol[kk]-1;
>>> > > > >       }
>>> > > > >       if(i%5 == 0 && mpiid==0){
>>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid:
>>> %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
>>> > > > >       }
>>> > > > >       countcol=tcountcol;
>>> > > > >       if(i%5 == 0 && mpiid==0){
>>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>>> > > > >       }
>>> > > > >     ierr =
>>> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>>> > > > >       if(i%5 == 0 && mpiid==0){
>>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n
>>> mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
>>> > > > >       }
>>> > > > >   }
>>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the
>>> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>>> > > > >
>>> > > > >
>>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>>> > > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>> > > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble
>>> the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>>> > > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
>>> > > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
>>> > > > >
>>> > > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
>>> > > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
>>> > > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
>>> > > > >   ierr =
>>> EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
>>> > > > >
>>> > > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
>>> > > > >   tol = 1.e-8;
>>> > > > >   maxit = 10000000;
>>> > > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
>>> > > > >
>>> > > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
>>> > > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
>>> > > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
>>> %f\n",t2-t1);CHKERRQ(ierr);
>>> > > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of
>>> the method: %D\n",its);CHKERRQ(ierr);
>>> > > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
>>> %s\n\n",type);CHKERRQ(ierr);
>>> > > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
>>> eigenvalues: %D\n",nev);CHKERRQ(ierr);
>>> > > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition:
>>> tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
>>> > > > >
>>> > > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
>>> > > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
>>> > > > >   /*
>>> > > > >   EPSGetConverged(eps,&nconv);
>>> > > > >   if (nconv>0) {
>>> > > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
>>> > > > >     EPSIsHermitian(eps,&ishermitian);
>>> > > > >     for (i=0;i<nconv;i++) {
>>> > > > >       EPSGetEigenvector(eps,i,xr,xi);
>>> > > > >       VecView(xr,viewer);
>>> > > > > #if !defined(PETSC_USE_COMPLEX)
>>> > > > >       if (!ishermitian) { VecView(xi,viewer); }
>>> > > > > #endif
>>> > > > >     }
>>> > > > >     PetscViewerDestroy(&viewer);
>>> > > > >   }
>>> > > > >   */
>>> > > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
>>> > > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
>>> > > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
>>> > > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
>>> > > > >   ierr = SlepcFinalize();
>>> > > > >   return 0;
>>> > > > > }
>>> > > > >
>>> > > > > OUTPUT:
>>> > > > >  Time used to build the matrix: 1914.729022
>>> > > > > [0] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [6] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [182] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [100] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [60] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [140] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [120] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [162] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [20] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [80] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [40] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [7] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [185] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [101] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [61] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [142] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [130] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [166] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [21] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [81] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [44] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [9] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [187] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [105] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [62] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [143] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [132] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [169] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [24] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [84] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [45] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [10] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [188] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [109] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [63] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [148] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [133] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [170] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [25] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [86] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [47] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [11] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [190] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [111] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [64] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [156] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [134] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [171] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [26] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [91] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [48] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [12] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [192] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [112] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [65] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [157] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [135] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [174] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [28] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [13] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [194] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [113] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [66] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [158] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [136] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [160] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [29] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [92] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [53] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [14] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [195] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [114] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [67] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [141] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [137] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [161] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [32] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [93] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [54] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [15] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [196] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [115] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [68] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [144] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [138] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [163] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [33] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [94] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [55] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [17] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [198] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [116] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [69] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [145] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [139] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [164] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [34] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [95] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [56] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [1] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [199] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [119] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [70] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [146] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [121] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [165] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [35] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [99] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [59] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [2] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [180] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [102] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [71] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [147] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [122] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [167] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [36] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [82] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [41] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [3] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [181] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [103] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [72] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [149] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [123] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [168] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [37] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [83] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [42] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [4] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [183] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [104] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [73] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [150] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [124] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [172] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [38] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [85] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [43] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [5] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [184] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [106] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [74] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [151] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [125] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [173] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [39] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [87] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [46] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [8] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [186] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [107] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [75] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [152] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [126] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [175] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [22] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [88] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [49] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [16] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [189] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [108] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [76] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [153] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [127] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [176] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [23] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [89] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [50] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [18] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [191] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [110] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [77] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [154] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [128] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [177] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [27] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [90] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [51] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [19] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [193] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [117] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [78] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [155] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [129] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [178] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [30] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [96] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [52] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [197] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [118] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [79] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [159] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [131] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [179] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [31] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [97] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [57] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [98] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [58] MatStashScatterBegin_Private(): No of messages: 0
>>> > > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>> mallocs.
>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3238253 unneeded,493497 used
>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>>> space: 3197999 unneeded,533751 used
>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3243809 unneeded,487941 used
>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3243446 unneeded,488258 used
>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>>> > > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3237359 unneeded,494391 used
>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3245082 unneeded,486622 used
>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3217795 unneeded,513955 used
>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>>> space: 3208633 unneeded,523117 used
>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3209020 unneeded,522684 used
>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3216305 unneeded,515445 used
>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3201643 unneeded,530107 used
>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3234040 unneeded,497664 used
>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>>> > > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3243435 unneeded,488315 used
>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3249858 unneeded,481846 used
>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3207727 unneeded,524023 used
>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3220647 unneeded,511103 used
>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>> > > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3197939 unneeded,533811 used
>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage
>>> space: 3179441 unneeded,552309 used
>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>> storage space: 3197890 unneeded,533814 used
>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>> MatSetValues() is 0
>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>> > > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>> > > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>> > > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>> storage space: 3200381 unneeded,531369 used
>>> > > > >
>>> > > > > <ex1.c><inp.log>
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > <1_Warning.txt>
>>> >
>>> >
>>> > <ex1.c><makefile>
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/4f0e0565/attachment-0001.html>

From knepley at gmail.com  Fri May  8 07:35:40 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 07:35:40 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi9siu+AB1CsYm2TVwXymxbtRbpRSJZOM30bLrx9hpZLkg@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
	<8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
	<CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
	<CAMYG4GktzqB3-rWiOAcZjiAmqswqyeeXuK5SqF1rK6wjw_Hg-A@mail.gmail.com>
	<CAJa9Fi9siu+AB1CsYm2TVwXymxbtRbpRSJZOM30bLrx9hpZLkg@mail.gmail.com>
Message-ID: <CAMYG4Gn+H_Fx4hC7zo1fA4CNh_BJLW9rXxrcZweC8nxXy=zaOA@mail.gmail.com>

On Fri, May 8, 2015 at 7:20 AM, Vijay Gopal Chilkuri <
vijay.gopal.c at gmail.com> wrote:

> Hi matt,
>
> I'm not generating values on random processors.
> here is the relevant part of the code:
>
>
>>   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>>   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>>   for (i=Istart; i<Iend; i++) {
>>       tcountcol=0;
>>       iii=i+1;
>>       unit_l1_(
>>             l1,
>>             l2,
>>             ktyp,
>>             &iii,
>>             xjjxy,
>>             xjjz ,
>>             xtt  ,
>>             &tcountcol,
>>             &ntrou,
>>             tcol,
>>             val);
>>       for(kk=0;kk<52;kk++){
>>           value[kk] = val[kk];
>>           col[kk] = tcol[kk]-1;
>>       }
>>       countcol=tcountcol;
>>     ierr =
>> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>>   }
>
>
> As can be seen, I get the relevant rows attributed to each processor and
> only MatSetValues() for the rows owned by the corresponding processors.
> That's why it is surprising that MatAssemnly takes a long time !
>

MatAssemblyEnd() does not do anything if there is no communication. This
cannot be a bottleneck. It is
another error in timing. We can confirm this by looking at the -log_summary
output.

  Thanks,

     Matt


> On Fri, May 8, 2015 at 12:21 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Fri, May 8, 2015 at 3:27 AM, Vijay Gopal Chilkuri <
>> vijay.gopal.c at gmail.com> wrote:
>>
>>> Thanks for testing.
>>>
>>> My problem is not with MatSetValues(), it is
>>> MatAssemblyBegin()/MatAssemblyEnd() that takes forever to finish.
>>> MatSetValues() is quite fast as you noticed.
>>>
>>> Actually, unit_l1_ is the subroutine that generates the nonzero terms
>>> (per row) of my hamiltonian. For large matrices this should be time
>>> consuming if done on a single processor.
>>> This is the reason that i use a lot of processors to divide the work of
>>> calling unit_l1_ multiple times.
>>> For example, if I use 171 processors over 10 nodes, unit_l1_ finishes in
>>> about 40 min. It is after this step at the MatAssemblyBegin() and
>>> MatAssemblyEnd() that the
>>> PETSc code get's stuck.
>>>
>>
>> Are you generating values on a random process, or on the process that
>> owns the row? If you
>> generate them anywhere, then MatAssemblyEnd() can have a large amount of
>> data to
>> communicate.
>>
>>   Matt
>>
>>
>>> If it's not too much trouble, could you perhaps test with more
>>> processors ?
>>>
>>> thanks again.
>>>
>>> On Thu, May 7, 2015 at 9:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>
>>>>
>>>>   I built the code with gfortran and optimization on my mac and run it
>>>> on one process. It is spending all its time in the unit_l1_ routine and
>>>> almost no time in the MatSetValues() in fact if I comment out the
>>>> MatSetValues() line it runs just as slow as before. How can you know the
>>>> time is spend in the MatSetValues()? Have you timed it also with the
>>>> MatSetValues() time commented out?
>>>>
>>>>    Barry
>>>>
>>>> > On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <
>>>> vijay.gopal.c at gmail.com> wrote:
>>>> >
>>>> > You are right ! I am using the IRPF90 tool to write my program in
>>>> FORTRAN.
>>>> >
>>>> > To use IRPF90 you can clone the github repository and get the irpf90
>>>> binary from the bin directory.
>>>> > The code to generate the library that contains UNIT_L1_ can be found
>>>> at: https://github.com/v1j4y/slepc_version.git
>>>> > just clone this repository and do:
>>>> >
>>>> > make
>>>> > make irpf90.a
>>>> >
>>>> > That's it !
>>>> >
>>>> > I'm resending you the PETSc files along with the makefile. you'll
>>>> need the irpf90.a library to make the ex1 executable.
>>>> >
>>>> > Hopefully this time it'll work.
>>>> > Please let me know if something goes wrong.
>>>> >
>>>> > On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> >
>>>> >   Our stupid mail system blocked the attachment because it contained
>>>> irpf90.a   Is this a library generated with
>>>> https://github.com/scemama/irpf90 ?
>>>> > You need to email everything that goes into making that library
>>>> because it is unlikely your .a library will work on my machine. What is
>>>> this thing and can you just tar up the directory that builds it and I can
>>>> generate irpf90.a
>>>> >
>>>> >   Barry
>>>> >
>>>> > > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <
>>>> vijay.gopal.c at gmail.com> wrote:
>>>> > >
>>>> > > Ok so here goes.
>>>> > >
>>>> > > I've attached a tarball with a directory which contains the
>>>> required files to compile ex1.c
>>>> > > The UNIT_L1_ subroutine generates the nonzero elements of my
>>>> hamiltonian given a row, it is contained in the
>>>> > > irpf90.a library. The actual fortran code is quite ugly.
>>>> > > Please tell me if you are able to compile and run the codes.
>>>> > >
>>>> > > Thanks a lot,
>>>> > >  Vijay
>>>> > >
>>>> > > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> > >
>>>> > > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <
>>>> vijay.gopal.c at gmail.com> wrote:
>>>> > > >
>>>> > > > Please let me clarify,
>>>> > > >
>>>> > > > @Matt:
>>>> > > > I've followed the proper manner to rapid assembly.
>>>> > > > That is :
>>>> > > >   1. preallocation
>>>> > > >   2. nonzero ownership
>>>> > > > Despite of having done the proper preallocation (as you can see
>>>> from the output file),
>>>> > > > the matrix takes ages to assemble, whereas the same code works
>>>> for a larger matrix with (100,000,000) elements taking 2 min to assemble !
>>>> > > >
>>>> > > > @Barry:
>>>> > > > I am not touching any file in the code that I attachd.
>>>> > > > What i'm doing is that each processor will (according to it's
>>>> ownership range) get it's nonzero entries
>>>> > > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for
>>>> their respective rows.
>>>> > > > Again to be clear I"m NOT doing file I/O anywhere.
>>>> > >
>>>> > >   Oh sorry, where is the fortran code? I assumed the fortran code
>>>> was reading from a file.
>>>> > >
>>>> > >    Based on what you sent yes it should take little time; if you
>>>> send something I could run I could try to see why it would be slow.
>>>> > >
>>>> > >   Barry
>>>> > >
>>>> > > >
>>>> > > > Could you please have a look at the files attached.
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>> > > >
>>>> > > >   You have two hundred processes banging on the same file on the
>>>> same filesystem fighting with each other to read in the matrix. That's like
>>>> hiring two hundred guys to dig a ditch but making them share one shovel;
>>>> you won't get speed up, you'll get speed down.
>>>> > > >
>>>> > > >
>>>> > > >   Make a simple sequential PETSc program that reads in the matrix
>>>> and then store it with MatView() in binary format. Then in your parallel
>>>> program use MatLoad() to read the matrix in (it will take a few seconds to
>>>> read in the matrix in that case).
>>>> > > >
>>>> > > >   Barry
>>>> > > >
>>>> > > >
>>>> > > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <
>>>> vijay.gopal.c at gmail.com> wrote:
>>>> > > > >
>>>> > > > > Hi,
>>>> > > > >
>>>> > > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
>>>> > > > > I've successfully diagonalized 100 million dimensional matrix
>>>> in 2hours.
>>>> > > > >
>>>> > > > > But, the problem is when i try to diagonailze (similar) a
>>>> smaller metrix of
>>>> > > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes
>>>> running 20 processors each (similar to the 100 million case), the matrix
>>>> assembly itself
>>>> > > > > takes 2h !
>>>> > > > >
>>>> > > > > Could someone point out the mistakes that i'm making ?
>>>> > > > >
>>>> > > > > i attach the source code and the output with the mail.
>>>> > > > >
>>>> > > > > thanks,
>>>> > > > >  Vijay
>>>> > > > >
>>>> > > > >
>>>> > > > > #include <slepceps.h>
>>>> > > > > #include <petsctime.h>
>>>> > > > >
>>>> > > > > #undef __FUNCT__
>>>> > > > > #define __FUNCT__ "main"
>>>> > > > > int main(int argc,char **argv)
>>>> > > > > {
>>>> > > > >   Mat            A;           /* problem matrix */
>>>> > > > >   EPS            eps;         /* eigenproblem solver context */
>>>> > > > >   EPSType        type;
>>>> > > > >   PetscReal      error,tol,re,im;
>>>> > > > >   PetscScalar    kr,ki,value[52];
>>>> > > > >   Vec            xr,xi;
>>>> > > > >   PetscInt
>>>>  n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
>>>> > > > >   PetscInt       d_nz,o_nz;
>>>> > > > >   PetscLogDouble t1,t2,tt1,tt2;
>>>> > > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
>>>> > > > >   PetscErrorCode ierr;
>>>> > > > >   PetscScalar    eigr;
>>>> > > > >   PetscScalar    eigi;
>>>> > > > >   PetscScalar    * data;
>>>> > > > >   Vec          Vr,Vi;
>>>> > > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
>>>> > > > >   PetscViewer    viewer;
>>>> > > > >   PetscBool      ishermitian;
>>>> > > > >   int            mpiid;
>>>> > > > >   long int            kk,iii;
>>>> > > > >   long int            tcountcol,tcol[52];
>>>> > > > >   float          val[52];
>>>> > > > >   long int            ntrou=1;
>>>> > > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
>>>> 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
>>>> 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
>>>> > > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
>>>> 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19,
>>>> 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
>>>>  };
>>>> > > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
>>>> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
>>>> > > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
>>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>>> > > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
>>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>>> > > > >   double           xtt[52]   ={-1.0,0.,0.,
>>>> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
>>>> > > > >
>>>> > > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem,
>>>> n=%D\n\n",n);CHKERRQ(ierr);
>>>> > > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>>>> > > > > //ierr =
>>>> MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
>>>> > > > >   ierr =
>>>> MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>>>> > > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>>>> > > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
>>>> > > > >   ierr =
>>>> MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
>>>> > > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
>>>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>>>> > > > >   for (i=Istart; i<Iend; i++) {
>>>> > > > >       tcountcol=0;
>>>> > > > >       iii=i+1;
>>>> > > > >       if(i%5 == 0 && mpiid==0){
>>>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>>>> > > > >       }
>>>> > > > >       unit_l1_(
>>>> > > > >             l1,
>>>> > > > >             l2,
>>>> > > > >             ktyp,
>>>> > > > >             &iii,
>>>> > > > >             xjjxy,
>>>> > > > >             xjjz ,
>>>> > > > >             xtt  ,
>>>> > > > >             &tcountcol,
>>>> > > > >             &ntrou,
>>>> > > > >             tcol,
>>>> > > > >             val);
>>>> > > > >       for(kk=0;kk<52;kk++){
>>>> > > > >           value[kk] = val[kk];
>>>> > > > >           col[kk] = tcol[kk]-1;
>>>> > > > >       }
>>>> > > > >       if(i%5 == 0 && mpiid==0){
>>>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>>>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid:
>>>> %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
>>>> > > > >       }
>>>> > > > >       countcol=tcountcol;
>>>> > > > >       if(i%5 == 0 && mpiid==0){
>>>> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
>>>> > > > >       }
>>>> > > > >     ierr =
>>>> MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
>>>> > > > >       if(i%5 == 0 && mpiid==0){
>>>> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
>>>> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n
>>>> mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
>>>> > > > >       }
>>>> > > > >   }
>>>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the
>>>> matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >
>>>> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
>>>> > > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>> > > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble
>>>> the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
>>>> > > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
>>>> > > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
>>>> > > > >   ierr =
>>>> EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
>>>> > > > >   tol = 1.e-8;
>>>> > > > >   maxit = 10000000;
>>>> > > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used:
>>>> %f\n",t2-t1);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of
>>>> the method: %D\n",its);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method:
>>>> %s\n\n",type);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested
>>>> eigenvalues: %D\n",nev);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
>>>> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition:
>>>> tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
>>>> > > > >
>>>> > > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
>>>> > > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
>>>> > > > >   /*
>>>> > > > >   EPSGetConverged(eps,&nconv);
>>>> > > > >   if (nconv>0) {
>>>> > > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
>>>> > > > >     EPSIsHermitian(eps,&ishermitian);
>>>> > > > >     for (i=0;i<nconv;i++) {
>>>> > > > >       EPSGetEigenvector(eps,i,xr,xi);
>>>> > > > >       VecView(xr,viewer);
>>>> > > > > #if !defined(PETSC_USE_COMPLEX)
>>>> > > > >       if (!ishermitian) { VecView(xi,viewer); }
>>>> > > > > #endif
>>>> > > > >     }
>>>> > > > >     PetscViewerDestroy(&viewer);
>>>> > > > >   }
>>>> > > > >   */
>>>> > > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
>>>> > > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
>>>> > > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
>>>> > > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
>>>> > > > >   ierr = SlepcFinalize();
>>>> > > > >   return 0;
>>>> > > > > }
>>>> > > > >
>>>> > > > > OUTPUT:
>>>> > > > >  Time used to build the matrix: 1914.729022
>>>> > > > > [0] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [6] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [182] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [100] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [60] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [140] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [120] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [162] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [20] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [80] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [40] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [7] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [185] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [101] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [61] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [142] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [130] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [166] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [21] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [81] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [44] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [9] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [187] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [105] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [62] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [143] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [132] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [169] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [24] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [84] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [45] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [10] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [188] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [109] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [63] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [148] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [133] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [170] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [25] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [86] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [47] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [11] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [190] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [111] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [64] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [156] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [134] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [171] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [26] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [91] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [48] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [12] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [192] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [112] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [65] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [157] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [135] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [174] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [28] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [13] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [194] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [113] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [66] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [158] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [136] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [160] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [29] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [92] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [53] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [14] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [195] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [114] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [67] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [141] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [137] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [161] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [32] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [93] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [54] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [15] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [196] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [115] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [68] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [144] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [138] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [163] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [33] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [94] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [55] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [17] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [198] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [116] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [69] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [145] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [139] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [164] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [34] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [95] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [56] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [1] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [199] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [119] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [70] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [146] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [121] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [165] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [35] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [99] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [59] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [2] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [180] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [102] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [71] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [147] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [122] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [167] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [36] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [82] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [41] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [3] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [181] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [103] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [72] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [149] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [123] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [168] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [37] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [83] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [42] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [4] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [183] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [104] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [73] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [150] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [124] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [172] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [38] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [85] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [43] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [5] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [184] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [106] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [74] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [151] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [125] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [173] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [39] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [87] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [46] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [8] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [186] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [107] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [75] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [152] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [126] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [175] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [22] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [88] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [49] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [16] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [189] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [108] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [76] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [153] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [127] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [176] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [23] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [89] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [50] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [18] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [191] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [110] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [77] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [154] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [128] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [177] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [27] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [90] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [51] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [19] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [193] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [117] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [78] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [155] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [129] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [178] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [30] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [96] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [52] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [197] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [118] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [79] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [159] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [131] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [179] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [31] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [97] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [57] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [98] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [58] MatStashScatterBegin_Private(): No of messages: 0
>>>> > > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0
>>>> mallocs.
>>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3238253 unneeded,493497 used
>>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3197999 unneeded,533751 used
>>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3243809 unneeded,487941 used
>>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3243446 unneeded,488258 used
>>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>>>> > > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3237359 unneeded,494391 used
>>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3245082 unneeded,486622 used
>>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3217795 unneeded,513955 used
>>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3208633 unneeded,523117 used
>>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3209020 unneeded,522684 used
>>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3216305 unneeded,515445 used
>>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3201643 unneeded,530107 used
>>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3234040 unneeded,497664 used
>>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
>>>> > > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3243435 unneeded,488315 used
>>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3249858 unneeded,481846 used
>>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3207727 unneeded,524023 used
>>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3220647 unneeded,511103 used
>>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
>>>> > > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3197939 unneeded,533811 used
>>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3179441 unneeded,552309 used
>>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124;
>>>> storage space: 3197890 unneeded,533814 used
>>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during
>>>> MatSetValues() is 0
>>>> > > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
>>>> > > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows
>>>> 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
>>>> > > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125;
>>>> storage space: 3200381 unneeded,531369 used
>>>> > > > >
>>>> > > > > <ex1.c><inp.log>
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > <1_Warning.txt>
>>>> >
>>>> >
>>>> > <ex1.c><makefile>
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/10f45db7/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri May  8 07:38:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 8 May 2015 07:38:46 -0500
Subject: [petsc-users] Matrix Assembly too slow
In-Reply-To: <CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
References: <CAJa9Fi_uRN3csS4RHBepRumzPAZsja7PLePyD8uB65LPH4qgAQ@mail.gmail.com>
	<9D6F1CDB-1B3A-49CD-8503-FD060098FD62@mcs.anl.gov>
	<CAJa9Fi-jDdmvS7uPsC=571sFEAL_reFwvqa=EznnV-jy8qSoNw@mail.gmail.com>
	<C43AACFD-DCEC-4534-9114-EC77D54DA8A4@mcs.anl.gov>
	<CAJa9Fi8VEVT+Kuv53=GwLNVKbSgPYJMN1v7S10q00P-qUOVcYQ@mail.gmail.com>
	<4FC10665-15A8-4CE8-AB2E-837381632AB8@mcs.anl.gov>
	<CAJa9Fi8q1+CLjpSxh-71RbEfFgPqYfkyK-nSOhy6p-1HLQPgxg@mail.gmail.com>
	<8D04D4A8-1C80-444A-95E4-491036FAAF4B@mcs.anl.gov>
	<CAJa9Fi_PKZ=Lrf+BjTrQ18sj2GB8u-dNvUEXDvyuO5JUtuHepA@mail.gmail.com>
Message-ID: <6CF664C9-4D80-43F9-B986-4A8E1CE3A6FA@mcs.anl.gov>


   MatAssemblyBegin/End in this case should take virtually no time; since you are not communicating any values it is only a barrier. This means all processes have to get to the MatAssemblyBegin/End before the program continues. So my guess is some processes get there much earlier than others so it looks like MatAssemblyBegin/End() takes forever but there is one process where it takes almost no time so it is a question of load balance.

  Barry

> On May 8, 2015, at 3:27 AM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> Thanks for testing.
> 
> My problem is not with MatSetValues(), it is MatAssemblyBegin()/MatAssemblyEnd() that takes forever to finish.
> MatSetValues() is quite fast as you noticed.
> 
> Actually, unit_l1_ is the subroutine that generates the nonzero terms (per row) of my hamiltonian. For large matrices this should be time consuming if done on a single processor. 
> This is the reason that i use a lot of processors to divide the work of calling unit_l1_ multiple times.
> For example, if I use 171 processors over 10 nodes, unit_l1_ finishes in about 40 min. It is after this step at the MatAssemblyBegin() and MatAssemblyEnd() that the 
> PETSc code get's stuck.
> 
> If it's not too much trouble, could you perhaps test with more processors ?
> 
> thanks again.
> 
> On Thu, May 7, 2015 at 9:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   I built the code with gfortran and optimization on my mac and run it on one process. It is spending all its time in the unit_l1_ routine and almost no time in the MatSetValues() in fact if I comment out the MatSetValues() line it runs just as slow as before. How can you know the time is spend in the MatSetValues()? Have you timed it also with the MatSetValues() time commented out?
> 
>    Barry
> 
> > On May 2, 2015, at 5:58 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> >
> > You are right ! I am using the IRPF90 tool to write my program in FORTRAN.
> >
> > To use IRPF90 you can clone the github repository and get the irpf90 binary from the bin directory.
> > The code to generate the library that contains UNIT_L1_ can be found at: https://github.com/v1j4y/slepc_version.git
> > just clone this repository and do:
> >
> > make
> > make irpf90.a
> >
> > That's it !
> >
> > I'm resending you the PETSc files along with the makefile. you'll need the irpf90.a library to make the ex1 executable.
> >
> > Hopefully this time it'll work.
> > Please let me know if something goes wrong.
> >
> > On Sun, May 3, 2015 at 12:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Our stupid mail system blocked the attachment because it contained  irpf90.a   Is this a library generated with https://github.com/scemama/irpf90 ?
> > You need to email everything that goes into making that library because it is unlikely your .a library will work on my machine. What is this thing and can you just tar up the directory that builds it and I can generate irpf90.a
> >
> >   Barry
> >
> > > On May 2, 2015, at 3:49 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > >
> > > Ok so here goes.
> > >
> > > I've attached a tarball with a directory which contains the required files to compile ex1.c
> > > The UNIT_L1_ subroutine generates the nonzero elements of my hamiltonian given a row, it is contained in the
> > > irpf90.a library. The actual fortran code is quite ugly.
> > > Please tell me if you are able to compile and run the codes.
> > >
> > > Thanks a lot,
> > >  Vijay
> > >
> > > On Sat, May 2, 2015 at 10:26 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > > > On May 2, 2015, at 2:57 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > > >
> > > > Please let me clarify,
> > > >
> > > > @Matt:
> > > > I've followed the proper manner to rapid assembly.
> > > > That is :
> > > >   1. preallocation
> > > >   2. nonzero ownership
> > > > Despite of having done the proper preallocation (as you can see from the output file),
> > > > the matrix takes ages to assemble, whereas the same code works for a larger matrix with (100,000,000) elements taking 2 min to assemble !
> > > >
> > > > @Barry:
> > > > I am not touching any file in the code that I attachd.
> > > > What i'm doing is that each processor will (according to it's ownership range) get it's nonzero entries
> > > > using a Fortran subroutine (UNIT_L1_) and call MatSetValues() for their respective rows.
> > > > Again to be clear I"m NOT doing file I/O anywhere.
> > >
> > >   Oh sorry, where is the fortran code? I assumed the fortran code was reading from a file.
> > >
> > >    Based on what you sent yes it should take little time; if you send something I could run I could try to see why it would be slow.
> > >
> > >   Barry
> > >
> > > >
> > > > Could you please have a look at the files attached.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, May 2, 2015 at 9:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > >
> > > >   You have two hundred processes banging on the same file on the same filesystem fighting with each other to read in the matrix. That's like hiring two hundred guys to dig a ditch but making them share one shovel; you won't get speed up, you'll get speed down.
> > > >
> > > >
> > > >   Make a simple sequential PETSc program that reads in the matrix and then store it with MatView() in binary format. Then in your parallel program use MatLoad() to read the matrix in (it will take a few seconds to read in the matrix in that case).
> > > >
> > > >   Barry
> > > >
> > > >
> > > > > On May 2, 2015, at 2:10 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm trying to diagonalize large matrices using PETSc and SLEPc.
> > > > > I've successfully diagonalized 100 million dimensional matrix in 2hours.
> > > > >
> > > > > But, the problem is when i try to diagonailze (similar) a smaller metrix of
> > > > > dimension 67,000,000 (24 nonzero elements per row) with 10 noes running 20 processors each (similar to the 100 million case), the matrix assembly itself
> > > > > takes 2h !
> > > > >
> > > > > Could someone point out the mistakes that i'm making ?
> > > > >
> > > > > i attach the source code and the output with the mail.
> > > > >
> > > > > thanks,
> > > > >  Vijay
> > > > >
> > > > >
> > > > > #include <slepceps.h>
> > > > > #include <petsctime.h>
> > > > >
> > > > > #undef __FUNCT__
> > > > > #define __FUNCT__ "main"
> > > > > int main(int argc,char **argv)
> > > > > {
> > > > >   Mat            A;           /* problem matrix */
> > > > >   EPS            eps;         /* eigenproblem solver context */
> > > > >   EPSType        type;
> > > > >   PetscReal      error,tol,re,im;
> > > > >   PetscScalar    kr,ki,value[52];
> > > > >   Vec            xr,xi;
> > > > >   PetscInt       n=16224936,ii,i,veclen,j,Istart,Iend,col[52],nev,maxit,its,nconv,countcol;
> > > > >   PetscInt       d_nz,o_nz;
> > > > >   PetscLogDouble t1,t2,tt1,tt2;
> > > > >   PetscBool      FirstBlock=PETSC_FALSE,LastBlock=PETSC_FALSE;
> > > > >   PetscErrorCode ierr;
> > > > >   PetscScalar    eigr;
> > > > >   PetscScalar    eigi;
> > > > >   PetscScalar    * data;
> > > > >   Vec          Vr,Vi;
> > > > >   char           filename[PETSC_MAX_PATH_LEN]="FIL666";
> > > > >   PetscViewer    viewer;
> > > > >   PetscBool      ishermitian;
> > > > >   int            mpiid;
> > > > >   long int            kk,iii;
> > > > >   long int            tcountcol,tcol[52];
> > > > >   float          val[52];
> > > > >   long int            ntrou=1;
> > > > >   long int            l1[52]={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > > >   long int            l2[52]={2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0       };
> > > > >   long int          ktyp[52]={1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0      };
> > > > >   double           xjjz[52]  ={0.0333333333333,-0.8, 0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >   double           xjjxy[52] ={0.0333333333333,-0.8,0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >   double           xtt[52]   ={-1.0,0.,0.,
> > > > >       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
> > > > >
> > > > >   SlepcInitialize(&argc,&argv,(char*)0,NULL);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD,"\n1-D t-J Eigenproblem, n=%D\n\n",n);CHKERRQ(ierr);
> > > > >   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
> > > > > //ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,n,n);CHKERRQ(ierr);
> > > > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > > >   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
> > > > >   ierr = MatSetUp(A);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);
> > > > >   ierr = MatCreateAIJ(PETSC_COMM_WORLD,Iend-Istart,Iend-Istart,n,n,46,NULL,46,NULL,&A);CHKERRQ(ierr);
> > > > >   MPI_Comm_rank(MPI_COMM_WORLD,&mpiid);
> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > > >   for (i=Istart; i<Iend; i++) {
> > > > >       tcountcol=0;
> > > > >       iii=i+1;
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >       }
> > > > >       unit_l1_(
> > > > >             l1,
> > > > >             l2,
> > > > >             ktyp,
> > > > >             &iii,
> > > > >             xjjxy,
> > > > >             xjjz ,
> > > > >             xtt  ,
> > > > >             &tcountcol,
> > > > >             &ntrou,
> > > > >             tcol,
> > > > >             val);
> > > > >       for(kk=0;kk<52;kk++){
> > > > >           value[kk] = val[kk];
> > > > >           col[kk] = tcol[kk]-1;
> > > > >       }
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," i: %d\n mpiid: %d\ntime: %f\n",i,mpiid,t2-t1);CHKERRQ(ierr);
> > > > >       }
> > > > >       countcol=tcountcol;
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >       }
> > > > >     ierr = MatSetValues(A,1,&i,countcol,col,value,INSERT_VALUES);CHKERRQ(ierr);
> > > > >       if(i%5 == 0 && mpiid==0){
> > > > >         ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >         ierr = PetscPrintf(PETSC_COMM_WORLD," processor \n mpiid: %d\ntime: %f\n",mpiid,t2-t1);CHKERRQ(ierr);
> > > > >       }
> > > > >   }
> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to build the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > > >
> > > > >
> > > > >   ierr = PetscTime(&tt1);CHKERRQ(ierr);
> > > > >   ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > > >   ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
> > > > >   ierr = PetscTime(&tt2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used to assemble the matrix: %f\n",tt2-tt1);CHKERRQ(ierr);
> > > > >   ierr = MatGetVecs(A,NULL,&xr);CHKERRQ(ierr);
> > > > >   ierr = MatGetVecs(A,NULL,&xi);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSCreate(PETSC_COMM_WORLD,&eps);CHKERRQ(ierr);
> > > > >   ierr = EPSSetOperators(eps,A,NULL);CHKERRQ(ierr);
> > > > >   ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
> > > > >   ierr = EPSSetWhichEigenpairs(eps,EPS_SMALLEST_REAL);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSSetFromOptions(eps);CHKERRQ(ierr);
> > > > >   tol = 1.e-8;
> > > > >   maxit = 10000000;
> > > > >   ierr = EPSSetTolerances(eps,tol,maxit);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = PetscTime(&t1);CHKERRQ(ierr);
> > > > >   ierr = EPSSolve(eps);CHKERRQ(ierr);
> > > > >   ierr = PetscTime(&t2);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Time used: %f\n",t2-t1);CHKERRQ(ierr);
> > > > >   ierr = EPSGetIterationNumber(eps,&its);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of iterations of the method: %D\n",its);CHKERRQ(ierr);
> > > > >   ierr = EPSGetType(eps,&type);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Solution method: %s\n\n",type);CHKERRQ(ierr);
> > > > >   ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Number of requested eigenvalues: %D\n",nev);CHKERRQ(ierr);
> > > > >   ierr = EPSGetTolerances(eps,&tol,&maxit);CHKERRQ(ierr);
> > > > >   ierr = PetscPrintf(PETSC_COMM_WORLD," Stopping condition: tol=%.4g, maxit=%D\n",(double)tol,maxit);CHKERRQ(ierr);
> > > > >
> > > > >   ierr = EPSGetConverged(eps,&nconv);CHKERRQ(ierr);
> > > > >   ierr = EPSPrintSolution(eps,NULL);CHKERRQ(ierr);
> > > > >   /*
> > > > >   EPSGetConverged(eps,&nconv);
> > > > >   if (nconv>0) {
> > > > >     PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&viewer);
> > > > >     EPSIsHermitian(eps,&ishermitian);
> > > > >     for (i=0;i<nconv;i++) {
> > > > >       EPSGetEigenvector(eps,i,xr,xi);
> > > > >       VecView(xr,viewer);
> > > > > #if !defined(PETSC_USE_COMPLEX)
> > > > >       if (!ishermitian) { VecView(xi,viewer); }
> > > > > #endif
> > > > >     }
> > > > >     PetscViewerDestroy(&viewer);
> > > > >   }
> > > > >   */
> > > > >   ierr = EPSDestroy(&eps);CHKERRQ(ierr);
> > > > >   ierr = MatDestroy(&A);CHKERRQ(ierr);
> > > > >   ierr = VecDestroy(&xr);CHKERRQ(ierr);
> > > > >   ierr = VecDestroy(&xi);CHKERRQ(ierr);
> > > > >   ierr = SlepcFinalize();
> > > > >   return 0;
> > > > > }
> > > > >
> > > > > OUTPUT:
> > > > >  Time used to build the matrix: 1914.729022
> > > > > [0] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [6] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [6] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [182] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [182] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [100] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [100] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [60] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [60] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [140] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [140] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [120] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [120] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [162] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [162] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [20] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [20] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [80] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [80] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [40] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [40] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [7] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [185] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [185] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [101] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [101] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [61] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [61] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [142] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [142] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [130] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [130] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [166] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [166] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [21] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [21] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [81] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [81] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [44] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [44] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [9] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [9] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [187] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [187] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [105] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [105] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [62] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [62] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [143] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [143] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [132] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [132] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [169] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [169] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [24] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [24] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [84] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [84] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [45] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [45] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [10] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [10] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [188] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [188] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [109] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [109] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [63] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [63] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [148] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [148] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [133] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [133] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [170] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [170] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [25] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [25] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [86] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [86] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [47] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [47] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [11] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [11] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [190] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [190] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [111] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [111] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [64] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [64] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [156] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [156] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [134] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [134] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [171] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [171] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [26] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [26] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [91] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [91] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [48] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [48] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [12] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [12] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [192] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [192] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [112] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [112] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [65] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [65] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [157] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [157] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [135] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [135] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [174] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [174] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [28] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [28] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [13] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [13] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [194] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [194] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [113] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [113] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [66] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [66] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [158] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [158] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [136] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [136] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [160] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [160] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [29] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [29] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [92] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [92] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [53] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [53] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [14] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [14] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [195] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [195] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [114] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [114] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [67] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [67] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [141] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [141] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [137] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [137] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [161] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [161] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [32] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [32] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [93] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [93] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [54] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [54] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [15] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [15] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [196] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [196] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [115] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [115] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [68] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [68] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [144] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [144] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [138] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [138] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [163] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [163] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [33] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [33] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [94] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [94] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [55] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [55] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [17] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [17] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [198] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [198] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [116] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [116] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [69] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [69] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [145] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [145] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [139] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [139] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [164] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [164] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [34] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [34] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [95] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [95] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [56] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [56] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [1] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [199] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [199] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [119] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [119] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [70] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [70] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [146] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [146] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [121] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [121] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [165] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [165] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [35] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [35] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [99] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [99] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [59] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [59] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [2] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [180] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [180] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [102] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [102] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [71] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [71] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [147] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [147] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [122] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [122] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [167] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [167] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [36] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [36] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [82] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [82] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [41] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [41] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [3] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [181] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [181] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [103] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [103] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [72] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [72] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [149] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [149] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [123] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [123] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [168] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [168] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [37] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [37] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [83] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [83] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [42] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [42] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [4] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [183] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [183] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [104] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [104] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [73] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [73] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [150] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [150] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [124] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [124] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [172] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [172] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [38] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [38] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [85] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [85] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [43] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [43] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [5] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [184] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [184] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [106] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [106] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [74] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [74] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [151] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [151] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [125] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [125] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [173] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [173] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [39] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [39] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [87] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [87] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [46] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [46] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [8] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [8] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [186] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [186] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [107] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [107] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [75] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [75] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [152] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [152] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [126] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [126] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [175] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [175] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [22] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [22] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [88] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [88] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [49] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [49] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [16] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [16] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [189] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [189] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [108] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [108] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [76] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [76] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [153] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [153] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [127] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [127] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [176] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [176] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [23] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [23] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [89] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [89] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [50] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [50] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [18] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [18] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [191] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [191] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [110] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [110] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [77] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [77] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [154] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [154] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [128] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [128] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [177] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [177] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [27] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [27] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [90] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [90] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [51] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [51] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [19] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [19] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [193] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [193] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [117] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [117] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [78] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [78] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [155] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [155] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [129] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [129] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [178] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [178] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [30] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [30] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [96] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [96] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [52] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [52] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [197] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [197] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [118] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [118] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [79] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [79] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [159] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [159] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [131] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [131] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [179] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [179] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [31] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [31] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [97] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [97] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [57] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [57] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [98] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [98] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [58] MatStashScatterBegin_Private(): No of messages: 0
> > > > > [58] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3238253 unneeded,493497 used
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [104] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [104] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197999 unneeded,533751 used
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243809 unneeded,487941 used
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [106] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [106] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3243446 unneeded,488258 used
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [144] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > > [144] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3237359 unneeded,494391 used
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [123] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [123] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3245082 unneeded,486622 used
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [173] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [173] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3217795 unneeded,513955 used
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [85] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [85] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3208633 unneeded,523117 used
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [6] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3209020 unneeded,522684 used
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [185] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [185] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3216305 unneeded,515445 used
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [108] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3201643 unneeded,530107 used
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [73] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [73] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3234040 unneeded,497664 used
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [146] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 11
> > > > > [146] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3243435 unneeded,488315 used
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [127] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [127] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3249858 unneeded,481846 used
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [177] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [177] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3207727 unneeded,524023 used
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [27] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [27] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3220647 unneeded,511103 used
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [87] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 12
> > > > > [87] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3197939 unneeded,533811 used
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [54] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [54] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3179441 unneeded,552309 used
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [8] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [8] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Matrix size: 81124 X 81124; storage space: 3197890 unneeded,533814 used
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > > > > [187] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 13
> > > > > [187] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81124) < 0.6. Do not use CompressedRow routines.
> > > > > [108] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 81125) < 0.6. Do not use CompressedRow routines.
> > > > > [74] MatAssemblyEnd_SeqAIJ(): Matrix size: 81125 X 81125; storage space: 3200381 unneeded,531369 used
> > > > >
> > > > > <ex1.c><inp.log>
> > > >
> > > >
> > >
> > >
> > > <1_Warning.txt>
> >
> >
> > <ex1.c><makefile>
> 
> 


From knepley at gmail.com  Fri May  8 07:40:32 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 07:40:32 -0500
Subject: [petsc-users] Structured grid in DMPlex?
In-Reply-To: <CAP2=TMiBe6FqFCKdyvHi6P6LNWL_OSt3EuDAsWy+-ab3KQmfNw@mail.gmail.com>
References: <CAP2=TMiBe6FqFCKdyvHi6P6LNWL_OSt3EuDAsWy+-ab3KQmfNw@mail.gmail.com>
Message-ID: <CAMYG4G=wsmQksVHSwpvE6WOz1zWkWPjdvcS9v2XwYQKV3atT8g@mail.gmail.com>

On Thu, May 7, 2015 at 10:31 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Hello everyone,
>
> If I know for a fact that I will be using structured grids AND still want
> the DMPlex data structure, what's the best/fastest way of achieving this?
> Should I:
>
> a) Invoke DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE,
> DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,-3,-3,PETSC_DECIDE,PETSC_DECIDE,1,1,NULL,NULL,&da);
> to get the parallel cell-vertex layout and (somehow) translate this into a
> DMPlex?
>
> b) Manually set the local DMPlex chart/DAG and assign the SF along the
> appropriate processors?
>
> c) Invoke some DMDA/DMPlex function or set of functions that already exist
> for this purpose?
>

It was supposed to work the same way as DMPlex in that you create the DM
and a DS, and then
the Section is created automatically. However, there is so much DA specific
stuff coded and no
one really wanted to use it, so its not maintained or consistent.


> d) Something else?
>

RIght now, I think the best thing to do is use DMPlexCreateHexBoxMesh(),
which gives you a structured mesh. Get
everything in your code working. Then, if you really feel limited by the
mesh structure, we can make the DMDA code
work.

  Thanks,

     Matt


> I know I could just input or create a coarse DAG and invoke -dm_refine,
> but I was wondering if there is an alternative to this method.
>
> Thanks,
> Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/605b07b7/attachment.html>

From knepley at gmail.com  Fri May  8 07:43:05 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 07:43:05 -0500
Subject: [petsc-users] DMPlex and HDF5 vector ordering
In-Reply-To: <CAP2=TMitVLADVp9=j5emnHx2AVegi24MG2JXGgf92Jys=C1adQ@mail.gmail.com>
References: <554C5681.7020704@auckland.ac.nz>
	<CAP2=TMitVLADVp9=j5emnHx2AVegi24MG2JXGgf92Jys=C1adQ@mail.gmail.com>
Message-ID: <CAMYG4Gk5_v1A+PK=ANy0v=4piUEQXLCBgdAOo=-jU+nm66JskQ@mail.gmail.com>

On Fri, May 8, 2015 at 1:48 AM, Justin Chang <jychang48 at gmail.com> wrote:

> I also had the same issue. My current work around is the following.
>
> 1) Run the first DMPlex program on one process and write the vector into
> HDF5.
>
> 2) Run the second DMPlex program with any number of processes but do the
> following:
>
> 3) After you create the initial DMPlex on rank 0, but before distributing
> it, duplicate it and create its petscsection and vector.
>
> 4) Load the HDF5 file into that vector. At this point the ordering is the
> same.
>
> 5) Distribute the original DM and save the PetscSF.
>
> 6) Call DMPlexDistributeField() to distribute the vector.
>
>
> This will guarantee the right ordering for the second program no matter
> how many processes it uses. Only drawback is that the first program has to
> be run in serial. I am also looking for a better way. Matt any thoughts?
>

Ata and Blaise have a pull request coming that creates a "natural ordering"
for a Plex, similar to the
one used by DMDA, so you get output that is invariant to process number. It
may take until the end
of the summer to get it fully integrated, but it is very close.

  Thanks,

     Matt


> Thanks,
> Justin
>
>
> On Friday, May 8, 2015, Adrian Croucher <a.croucher at auckland.ac.nz> wrote:
>
>> hi,
>>
>> I create a Vec on a DMPlex using DMPlexCreateGlobalVec(), then write it
>> to HDF5 using PetscViewerHDF5Open() and VecView().
>>
>> I then try to read it back in later (in another program, but using the
>> same DMPlex) using PetscViewerHDF5Open() and VecLoad().
>>
>> It looks like the ordering of the final vector entries in the second
>> program depends on how many processors I use. If they are the same in both
>> programs, I get the right ordering, but if they aren't, I don't. Is that
>> expected? If so, is there any way to guarantee the right ordering when I
>> read the Vec back in?
>>
>> - Adrian
>>
>> --
>> Dr Adrian Croucher
>> Senior Research Fellow
>> Department of Engineering Science
>> University of Auckland, New Zealand
>> email: a.croucher at auckland.ac.nz
>> tel: +64 (0)9 923 84611
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/245f5615/attachment.html>

From jychang48 at gmail.com  Fri May  8 08:08:23 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 8 May 2015 08:08:23 -0500
Subject: [petsc-users] Structured grid in DMPlex?
In-Reply-To: <CAMYG4G=wsmQksVHSwpvE6WOz1zWkWPjdvcS9v2XwYQKV3atT8g@mail.gmail.com>
References: <CAP2=TMiBe6FqFCKdyvHi6P6LNWL_OSt3EuDAsWy+-ab3KQmfNw@mail.gmail.com>
	<CAMYG4G=wsmQksVHSwpvE6WOz1zWkWPjdvcS9v2XwYQKV3atT8g@mail.gmail.com>
Message-ID: <CAP2=TMisdKL9CFpFPRSTw0iCF1X27cTXj-ek7T5d8FRduu5Nmw@mail.gmail.com>

Matt,

Creating structured meshes on rank 0 is no problem. The issue I am having
is the amount of time it takes to distribute the mesh across multiple
processes. If I know the grid has a structured format, then wouldn't it be
much faster to have each process create its local portion of the DMPlex
chart instead of calling a graph partitioner? As long as we know the global
number of cells/points in each spatial direction, the desired local
organization of cells/points for each rank, and where the ghost nodes
reside, I imagine this should be (almost) entirely local and scalable. I
would think the only real work that goes into this is to determine a prior
an algorithm that evenly distributes the mesh points and to create the SF
associate with each local portion of the DM?

Thanks,
Justin

On Fri, May 8, 2015 at 7:40 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, May 7, 2015 at 10:31 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hello everyone,
>>
>> If I know for a fact that I will be using structured grids AND still want
>> the DMPlex data structure, what's the best/fastest way of achieving this?
>> Should I:
>>
>> a) Invoke DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE,
>> DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,-3,-3,PETSC_DECIDE,PETSC_DECIDE,1,1,NULL,NULL,&da);
>> to get the parallel cell-vertex layout and (somehow) translate this into a
>> DMPlex?
>>
>> b) Manually set the local DMPlex chart/DAG and assign the SF along the
>> appropriate processors?
>>
>> c) Invoke some DMDA/DMPlex function or set of functions that already
>> exist for this purpose?
>>
>
> It was supposed to work the same way as DMPlex in that you create the DM
> and a DS, and then
> the Section is created automatically. However, there is so much DA
> specific stuff coded and no
> one really wanted to use it, so its not maintained or consistent.
>
>
>> d) Something else?
>>
>
> RIght now, I think the best thing to do is use DMPlexCreateHexBoxMesh(),
> which gives you a structured mesh. Get
> everything in your code working. Then, if you really feel limited by the
> mesh structure, we can make the DMDA code
> work.
>
>   Thanks,
>
>      Matt
>
>
>> I know I could just input or create a coarse DAG and invoke -dm_refine,
>> but I was wondering if there is an alternative to this method.
>>
>> Thanks,
>> Justin
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/94eee3b5/attachment.html>

From mfadams at lbl.gov  Fri May  8 08:28:25 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 8 May 2015 09:28:25 -0400
Subject: [petsc-users] configure error on Titan with Intel
Message-ID: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>

I am getting this error on Titan with Intel programming env.

I run this from an interactive batch shell to avoid the batch business.

This workflow seemed to work for pgi & gnu programming envs.

Any ideas?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/b4f32cab/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 85447 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/b4f32cab/attachment-0001.obj>

From knepley at gmail.com  Fri May  8 08:48:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 08:48:39 -0500
Subject: [petsc-users] Structured grid in DMPlex?
In-Reply-To: <CAP2=TMisdKL9CFpFPRSTw0iCF1X27cTXj-ek7T5d8FRduu5Nmw@mail.gmail.com>
References: <CAP2=TMiBe6FqFCKdyvHi6P6LNWL_OSt3EuDAsWy+-ab3KQmfNw@mail.gmail.com>
	<CAMYG4G=wsmQksVHSwpvE6WOz1zWkWPjdvcS9v2XwYQKV3atT8g@mail.gmail.com>
	<CAP2=TMisdKL9CFpFPRSTw0iCF1X27cTXj-ek7T5d8FRduu5Nmw@mail.gmail.com>
Message-ID: <CAMYG4GmkF9B4tVx=0-rSvrkL+F5wFEFRr=srYO6eLZ0UPcYdjQ@mail.gmail.com>

On Fri, May 8, 2015 at 8:08 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Matt,
>
> Creating structured meshes on rank 0 is no problem. The issue I am having
> is the amount of time it takes to distribute the mesh across multiple
> processes. If I know the grid has a structured format, then wouldn't it be
> much faster to have each process create its local portion of the DMPlex
> chart instead of calling a graph partitioner? As long as we know the global
> number of cells/points in each spatial direction, the desired local
> organization of cells/points for each rank, and where the ghost nodes
> reside, I imagine this should be (almost) entirely local and scalable. I
> would think the only real work that goes into this is to determine a prior
> an algorithm that evenly distributes the mesh points and to create the SF
> associate with each local portion of the DM?
>

Ah, yes that is a bottleneck as you get larger. If that is the problem, for
now just use -dm_refine <num>.

  Thanks,

     Matt


> Thanks,
> Justin
>
> On Fri, May 8, 2015 at 7:40 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, May 7, 2015 at 10:31 PM, Justin Chang <jychang48 at gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> If I know for a fact that I will be using structured grids AND still
>>> want the DMPlex data structure, what's the best/fastest way of achieving
>>> this? Should I:
>>>
>>> a) Invoke DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE,
>>> DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,-3,-3,PETSC_DECIDE,PETSC_DECIDE,1,1,NULL,NULL,&da);
>>> to get the parallel cell-vertex layout and (somehow) translate this into a
>>> DMPlex?
>>>
>>> b) Manually set the local DMPlex chart/DAG and assign the SF along the
>>> appropriate processors?
>>>
>>> c) Invoke some DMDA/DMPlex function or set of functions that already
>>> exist for this purpose?
>>>
>>
>> It was supposed to work the same way as DMPlex in that you create the DM
>> and a DS, and then
>> the Section is created automatically. However, there is so much DA
>> specific stuff coded and no
>> one really wanted to use it, so its not maintained or consistent.
>>
>>
>>> d) Something else?
>>>
>>
>> RIght now, I think the best thing to do is use DMPlexCreateHexBoxMesh(),
>> which gives you a structured mesh. Get
>> everything in your code working. Then, if you really feel limited by the
>> mesh structure, we can make the DMDA code
>> work.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> I know I could just input or create a coarse DAG and invoke -dm_refine,
>>> but I was wondering if there is an alternative to this method.
>>>
>>> Thanks,
>>> Justin
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/140cf5d7/attachment.html>

From balay at mcs.anl.gov  Fri May  8 10:46:24 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 10:46:24 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505081044220.2737@asterix>

On Fri, 8 May 2015, Mark Adams wrote:

> I am getting this error on Titan with Intel programming env.
> 
> I run this from an interactive batch shell to avoid the batch business.
> 
> This workflow seemed to work for pgi & gnu programming envs.
> 
> Any ideas?

>>>>>
ERROR while running executable: Could not execute "/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest":
/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest: error while loading shared libraries: libacml.so: cannot open shared object file: No such file or directory
<<<<<<

Perhaps ACML module is not setup [or loaded] correctly for intel compilers.

You can manually set LD_LIBRARY_PATH to its location - and see if the
problem goes away.

Satish

---------

balay at asterix /home/balay/download-pine
$ grep LD_LIBRARY_PATH configure.log
CRAY_LD_LIBRARY_PATH=/opt/cray/pmi/5.0.5-1.0000.10300.134.8.gem/lib64:/opt/cray/libsci/13.0.1/INTEL/140/interlagos/lib:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib:/opt/cray/mpt/7.0.4/gni/mpich2-intel/140/lib:/opt/cray/alps/5.2.1-2.0502.9479.20.1.gem/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/extras/CUPTI/lib64:/opt/cray/netcdf-hdf5parallel/4.3.2/PGI/141/lib:/opt/cray/mpt/7.0.4/gni/mpich2-pgi/141/lib:/opt/cray/rca/1.0.0-2.0502.53711.3.125.gem/lib64:/opt/cray/xpmem/0.1-2.0502.55507.3.2.gem/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.211.gem/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.gem/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.25.gem/lib64:/opt/cray/libsci/13.0.1/PGI/141/x86_64/lib
MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.1:/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/lib:/lustre/atlas/sw/xk7/python/2.7.9/sles11.3_gnu4.3.4/lib:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/32:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/64
  OpenMPI wrappers require LD_LIBRARY_PATH set
balay at asterix /home/balay/download-pine
$ grep LD_LIBRARY_PATH configure.log |grep -i acml
balay at asterix /home/balay/download-pine
$ 

From jychang48 at gmail.com  Fri May  8 11:00:53 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Fri, 8 May 2015 11:00:53 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87h9rodxwj.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<87fv7cjxwc.fsf@jedbrown.org>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org> <87k2wkdyrl.fsf@jedbrown.org>
	<CAMYG4G=6T=dymx=UO=9iVJAG7UNh-_o9=8UU5BfExNVkKzQoHw@mail.gmail.com>
	<87h9rodxwj.fsf@jedbrown.org>
Message-ID: <CAP2=TMgrw2J9zmYDBdS3VMZ8XQFqvM_EfanLLDporKYzg7oOrg@mail.gmail.com>

For clarification purposes:

1) What is the definition of "performance model" and "cache model"? I see
those two terms used in this thread but want to know the exact difference
if any.

2) Is what's described in Dinesh's paper a "cache model"?
What exactly is the caveat or what are the major assumptions that it makes?

3) Is quantifying the "useful bandwidth sustained for some level of catch"
analogous/related to cache register reuse and/or vectorization (e.g., how
well one can maximize SIMD on the machine if that makes any sense)

Thank you guys for all your help,
Justin

On Thursday, May 7, 2015, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Thu, May 7, 2015 at 1:32 PM, Jed Brown <jed at jedbrown.org> wrote:
> >
> >> Matthew Knepley <knepley at gmail.com> writes:
> >> > You are making assumptions about the algorithm here. Pure streaming
> >> > computations, like VecAXPY do not depend on the cache model.
> >>
> >> I thought we were talking about Krylov solvers, which includes MatMult.
> >>
> >
> > Yes, and in the earlier mail I said to just use the model in Dinesh's
> > paper so he did not have to worry about this one.
>
> That is a cache model (analogous to "perfect cache" in the pTatin
> paper).  So don't claim you have no cache model.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/e2ce1ee0/attachment.html>

From mfadams at lbl.gov  Fri May  8 11:03:31 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 8 May 2015 12:03:31 -0400
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <alpine.LFD.2.11.1505081044220.2737@asterix>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
Message-ID: <CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>

ACML seems to be there and Ed D'Azevedo gave me a module load script that I
tried.  I see:

/usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm

And I also still get the failure to find libacml.so

I will try your LD_LIBRARY_PATH suggestion.


On Fri, May 8, 2015 at 11:46 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Fri, 8 May 2015, Mark Adams wrote:
>
> > I am getting this error on Titan with Intel programming env.
> >
> > I run this from an interactive batch shell to avoid the batch business.
> >
> > This workflow seemed to work for pgi & gnu programming envs.
> >
> > Any ideas?
>
> >>>>>
> ERROR while running executable: Could not execute
> "/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest":
> /tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest: error while
> loading shared libraries: libacml.so: cannot open shared object file: No
> such file or directory
> <<<<<<
>
> Perhaps ACML module is not setup [or loaded] correctly for intel compilers.
>
> You can manually set LD_LIBRARY_PATH to its location - and see if the
> problem goes away.
>
> Satish
>
> ---------
>
> balay at asterix /home/balay/download-pine
> $ grep LD_LIBRARY_PATH configure.log
>
> CRAY_LD_LIBRARY_PATH=/opt/cray/pmi/5.0.5-1.0000.10300.134.8.gem/lib64:/opt/cray/libsci/13.0.1/INTEL/140/interlagos/lib:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib:/opt/cray/mpt/7.0.4/gni/mpich2-intel/140/lib:/opt/cray/alps/5.2.1-2.0502.9479.20.1.gem/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/extras/CUPTI/lib64:/opt/cray/netcdf-hdf5parallel/4.3.2/PGI/141/lib:/opt/cray/mpt/7.0.4/gni/mpich2-pgi/141/lib:/opt/cray/rca/1.0.0-2.0502.53711.3.125.gem/lib64:/opt/cray/xpmem/0.1-2.0502.55507.3.2.gem/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.211.gem/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.gem/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.25.gem/lib64:/opt/cray/libsci/13.0.1/PGI/141/x86_64/lib
>
> MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
>
> LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.1:/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/lib:/lustre/atlas/sw/xk7/python/2.7.9/sles11.3_gnu4.3.4/lib:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/32:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/64
>   OpenMPI wrappers require LD_LIBRARY_PATH set
> balay at asterix /home/balay/download-pine
> $ grep LD_LIBRARY_PATH configure.log |grep -i acml
> balay at asterix /home/balay/download-pine
> $
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/74880826/attachment.html>

From balay at mcs.anl.gov  Fri May  8 11:09:40 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 11:09:40 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505081108330.2737@asterix>

On Fri, 8 May 2015, Mark Adams wrote:

> ACML seems to be there and Ed D'Azevedo gave me a module load script that I
> tried.  I see:
> 
> /usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm

This is expected behavior [this test is checking for broken compilers
that don't give errors here..]

Satish

> 
> And I also still get the failure to find libacml.so
> 
> I will try your LD_LIBRARY_PATH suggestion.
> 
> 
> On Fri, May 8, 2015 at 11:46 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Fri, 8 May 2015, Mark Adams wrote:
> >
> > > I am getting this error on Titan with Intel programming env.
> > >
> > > I run this from an interactive batch shell to avoid the batch business.
> > >
> > > This workflow seemed to work for pgi & gnu programming envs.
> > >
> > > Any ideas?
> >
> > >>>>>
> > ERROR while running executable: Could not execute
> > "/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest":
> > /tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest: error while
> > loading shared libraries: libacml.so: cannot open shared object file: No
> > such file or directory
> > <<<<<<
> >
> > Perhaps ACML module is not setup [or loaded] correctly for intel compilers.
> >
> > You can manually set LD_LIBRARY_PATH to its location - and see if the
> > problem goes away.
> >
> > Satish
> >
> > ---------
> >
> > balay at asterix /home/balay/download-pine
> > $ grep LD_LIBRARY_PATH configure.log
> >
> > CRAY_LD_LIBRARY_PATH=/opt/cray/pmi/5.0.5-1.0000.10300.134.8.gem/lib64:/opt/cray/libsci/13.0.1/INTEL/140/interlagos/lib:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib:/opt/cray/mpt/7.0.4/gni/mpich2-intel/140/lib:/opt/cray/alps/5.2.1-2.0502.9479.20.1.gem/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/extras/CUPTI/lib64:/opt/cray/netcdf-hdf5parallel/4.3.2/PGI/141/lib:/opt/cray/mpt/7.0.4/gni/mpich2-pgi/141/lib:/opt/cray/rca/1.0.0-2.0502.53711.3.125.gem/lib64:/opt/cray/xpmem/0.1-2.0502.55507.3.2.gem/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.211.gem/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.gem/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.25.gem/lib64:/opt/cray/libsci/13.0.1/PGI/141/x86_64/lib
> >
> > MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
> >
> > LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.1:/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/lib:/lustre/atlas/sw/xk7/python/2.7.9/sles11.3_gnu4.3.4/lib:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/32:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/64
> >   OpenMPI wrappers require LD_LIBRARY_PATH set
> > balay at asterix /home/balay/download-pine
> > $ grep LD_LIBRARY_PATH configure.log |grep -i acml
> > balay at asterix /home/balay/download-pine
> > $
> >
> 


From dazevedoef at ornl.gov  Fri May  8 11:16:30 2015
From: dazevedoef at ornl.gov (Ed D'Azevedo)
Date: Fri, 08 May 2015 12:16:30 -0400
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
Message-ID: <554CE15E.90109@ornl.gov>

Is petsc using ACML specific routines?
If not, perhaps MKL provides the lapack/blas and ACML is not needed?

Just a thought.

Ed


On 05/08/2015 12:03 PM, Mark Adams wrote:
> ACML seems to be there and Ed D'Azevedo gave me a module load script 
> that I tried.  I see:
>
> /usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
>
> And I also still get the failure to find libacml.so
>
> I will try your LD_LIBRARY_PATH suggestion.
>
>
> On Fri, May 8, 2015 at 11:46 AM, Satish Balay <balay at mcs.anl.gov 
> <mailto:balay at mcs.anl.gov>> wrote:
>
>     On Fri, 8 May 2015, Mark Adams wrote:
>
>     > I am getting this error on Titan with Intel programming env.
>     >
>     > I run this from an interactive batch shell to avoid the batch
>     business.
>     >
>     > This workflow seemed to work for pgi & gnu programming envs.
>     >
>     > Any ideas?
>
>     >>>>>
>     ERROR while running executable: Could not execute
>     "/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest":
>     /tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest: error
>     while loading shared libraries: libacml.so: cannot open shared
>     object file: No such file or directory
>     <<<<<<
>
>     Perhaps ACML module is not setup [or loaded] correctly for intel
>     compilers.
>
>     You can manually set LD_LIBRARY_PATH to its location - and see if the
>     problem goes away.
>
>     Satish
>
>     ---------
>
>     balay at asterix /home/balay/download-pine
>     $ grep LD_LIBRARY_PATH configure.log
>     CRAY_LD_LIBRARY_PATH=/opt/cray/pmi/5.0.5-1.0000.10300.134.8.gem/lib64:/opt/cray/libsci/13.0.1/INTEL/140/interlagos/lib:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib:/opt/cray/mpt/7.0.4/gni/mpich2-intel/140/lib:/opt/cray/alps/5.2.1-2.0502.9479.20.1.gem/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/extras/CUPTI/lib64:/opt/cray/netcdf-hdf5parallel/4.3.2/PGI/141/lib:/opt/cray/mpt/7.0.4/gni/mpich2-pgi/141/lib:/opt/cray/rca/1.0.0-2.0502.53711.3.125.gem/lib64:/opt/cray/xpmem/0.1-2.0502.55507.3.2.gem/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.211.gem/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.gem/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.25.gem/lib64:/opt/cray/libsci/13.0.1/PGI/141/x86_64/lib
>     MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
>     LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.1:/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/lib:/lustre/atlas/sw/xk7/python/2.7.9/sles11.3_gnu4.3.4/lib:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/32:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/64
>       OpenMPI wrappers require LD_LIBRARY_PATH set
>     balay at asterix /home/balay/download-pine
>     $ grep LD_LIBRARY_PATH configure.log |grep -i acml
>     balay at asterix /home/balay/download-pine
>     $
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/2b12e2e4/attachment.html>

From balay at mcs.anl.gov  Fri May  8 11:19:25 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 11:19:25 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <alpine.LFD.2.11.1505081108330.2737@asterix>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
Message-ID: <alpine.LFD.2.11.1505081115200.2737@asterix>

Also - Perhaps with intel compilers - the recommended blas is
something other than ACML? [like MKL?]. Something to check..

Satish

On Fri, 8 May 2015, Satish Balay wrote:

> On Fri, 8 May 2015, Mark Adams wrote:
> 
> > ACML seems to be there and Ed D'Azevedo gave me a module load script that I
> > tried.  I see:
> > 
> > /usr/bin/ld: cannot find -lpetsc-ufod4vtr9mqHvKIQiVAm
> 
> This is expected behavior [this test is checking for broken compilers
> that don't give errors here..]
> 
> Satish
> 
> > 
> > And I also still get the failure to find libacml.so
> > 
> > I will try your LD_LIBRARY_PATH suggestion.
> > 
> > 
> > On Fri, May 8, 2015 at 11:46 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > > On Fri, 8 May 2015, Mark Adams wrote:
> > >
> > > > I am getting this error on Titan with Intel programming env.
> > > >
> > > > I run this from an interactive batch shell to avoid the batch business.
> > > >
> > > > This workflow seemed to work for pgi & gnu programming envs.
> > > >
> > > > Any ideas?
> > >
> > > >>>>>
> > > ERROR while running executable: Could not execute
> > > "/tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest":
> > > /tmp/scratch/petsc-vW7iN1/config.setCompilers/conftest: error while
> > > loading shared libraries: libacml.so: cannot open shared object file: No
> > > such file or directory
> > > <<<<<<
> > >
> > > Perhaps ACML module is not setup [or loaded] correctly for intel compilers.
> > >
> > > You can manually set LD_LIBRARY_PATH to its location - and see if the
> > > problem goes away.
> > >
> > > Satish
> > >
> > > ---------
> > >
> > > balay at asterix /home/balay/download-pine
> > > $ grep LD_LIBRARY_PATH configure.log
> > >
> > > CRAY_LD_LIBRARY_PATH=/opt/cray/pmi/5.0.5-1.0000.10300.134.8.gem/lib64:/opt/cray/libsci/13.0.1/INTEL/140/interlagos/lib:/opt/cray/netcdf-hdf5parallel/4.3.2/INTEL/140/lib:/opt/cray/mpt/7.0.4/gni/mpich2-intel/140/lib:/opt/cray/alps/5.2.1-2.0502.9479.20.1.gem/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/lib64:/opt/nvidia/cudatoolkit/5.5.51-1.0502.9594.3.1/extras/CUPTI/lib64:/opt/cray/netcdf-hdf5parallel/4.3.2/PGI/141/lib:/opt/cray/mpt/7.0.4/gni/mpich2-pgi/141/lib:/opt/cray/rca/1.0.0-2.0502.53711.3.125.gem/lib64:/opt/cray/xpmem/0.1-2.0502.55507.3.2.gem/lib64:/opt/cray/dmapp/7.0.1-1.0502.9501.5.211.gem/lib64:/opt/cray/ugni/5.0-1.0502.9685.4.24.gem/lib64:/opt/cray/udreg/2.3.2-1.0502.9275.1.25.gem/lib64:/opt/cray/libsci/13.0.1/PGI/141/x86_64/lib
> > >
> > > MIC_LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/mic:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/mic
> > >
> > > LD_LIBRARY_PATH=/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.1:/sw/xk6/python/2.7.9/sles11.3_gnu4.3.4/lib:/lustre/atlas/sw/xk7/python/2.7.9/sles11.3_gnu4.3.4/lib:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/32:/opt/sw/xk6/ddt/4.2-35818/sles11.1_binary/lib/64
> > >   OpenMPI wrappers require LD_LIBRARY_PATH set
> > > balay at asterix /home/balay/download-pine
> > > $ grep LD_LIBRARY_PATH configure.log |grep -i acml
> > > balay at asterix /home/balay/download-pine
> > > $
> > >
> > 
> 
> 


From balay at mcs.anl.gov  Fri May  8 11:22:20 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 11:22:20 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <554CE15E.90109@ornl.gov>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
Message-ID: <alpine.LFD.2.11.1505081119570.2737@asterix>

On Fri, 8 May 2015, Ed D'Azevedo wrote:

> Is petsc using ACML specific routines?
> If not, perhaps MKL provides the lapack/blas and ACML is not needed?
> 
> Just a thought.

yup PETSc can use whatever blas/lapck user provides.

In this case PETSc was not seeking acml - but the 'module' env was
presumably setup with acml [and the compilers are attempting to use
these modules directly - irrespective of PETSc]

Satish

From dptrebotich at lbl.gov  Fri May  8 12:23:13 2015
From: dptrebotich at lbl.gov (David Trebotich)
Date: Fri, 8 May 2015 10:23:13 -0700
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <alpine.LFD.2.11.1505081119570.2737@asterix>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
	<alpine.LFD.2.11.1505081119570.2737@asterix>
Message-ID: <CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>

This has all come up because I cannot build Chombo-crunch using gnu
prgenv---cannot find mkl anymore.  So olcf consultants advised Intel
prgenv.  I am not sure yet what form of libs are used, that is mkl, blas, I
don't know.

David Trebotich
sent from mobile
(510) 384-6868
On May 8, 2015 9:22 AM, "Satish Balay" <balay at mcs.anl.gov> wrote:

> On Fri, 8 May 2015, Ed D'Azevedo wrote:
>
> > Is petsc using ACML specific routines?
> > If not, perhaps MKL provides the lapack/blas and ACML is not needed?
> >
> > Just a thought.
>
> yup PETSc can use whatever blas/lapck user provides.
>
> In this case PETSc was not seeking acml - but the 'module' env was
> presumably setup with acml [and the compilers are attempting to use
> these modules directly - irrespective of PETSc]
>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/94734f4e/attachment.html>

From mfadams at lbl.gov  Fri May  8 12:28:58 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 8 May 2015 13:28:58 -0400
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
	<alpine.LFD.2.11.1505081119570.2737@asterix>
	<CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
Message-ID: <CADOhEh4D+yEK4HTr_8qheGQ79g5ajZBWTqdNv+jQRy9jsQ3Ujw@mail.gmail.com>

I added the LD path stuff and that seems to have fixed this problem ... but
there are still problems.

On Fri, May 8, 2015 at 1:23 PM, David Trebotich <dptrebotich at lbl.gov> wrote:

> This has all come up because I cannot build Chombo-crunch using gnu
> prgenv---cannot find mkl anymore.  So olcf consultants advised Intel
> prgenv.  I am not sure yet what form of libs are used, that is mkl, blas, I
> don't know.
>
> David Trebotich
> sent from mobile
> (510) 384-6868
> On May 8, 2015 9:22 AM, "Satish Balay" <balay at mcs.anl.gov> wrote:
>
>> On Fri, 8 May 2015, Ed D'Azevedo wrote:
>>
>> > Is petsc using ACML specific routines?
>> > If not, perhaps MKL provides the lapack/blas and ACML is not needed?
>> >
>> > Just a thought.
>>
>> yup PETSc can use whatever blas/lapck user provides.
>>
>> In this case PETSc was not seeking acml - but the 'module' env was
>> presumably setup with acml [and the compilers are attempting to use
>> these modules directly - irrespective of PETSc]
>>
>> Satish
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/280561e8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 70168 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/280561e8/attachment-0001.obj>

From balay at mcs.anl.gov  Fri May  8 14:11:42 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 14:11:42 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
	<alpine.LFD.2.11.1505081119570.2737@asterix>
	<CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505081410240.2737@asterix>

On Fri, 8 May 2015, David Trebotich wrote:

> This has all come up because I cannot build Chombo-crunch using gnu
> prgenv---cannot find mkl anymore.

If this is a PETSc issue - send us logs - and I can look at it.

With gnu compilers one can use gfortran compiled blas/lapack [MKL
should also work - but it has to be specified correctly].

Satish

>  So olcf consultants advised Intel
> prgenv.  I am not sure yet what form of libs are used, that is mkl, blas, I
> don't know.
> 
> David Trebotich
> sent from mobile
> (510) 384-6868
> On May 8, 2015 9:22 AM, "Satish Balay" <balay at mcs.anl.gov> wrote:
> 
> > On Fri, 8 May 2015, Ed D'Azevedo wrote:
> >
> > > Is petsc using ACML specific routines?
> > > If not, perhaps MKL provides the lapack/blas and ACML is not needed?
> > >
> > > Just a thought.
> >
> > yup PETSc can use whatever blas/lapck user provides.
> >
> > In this case PETSc was not seeking acml - but the 'module' env was
> > presumably setup with acml [and the compilers are attempting to use
> > these modules directly - irrespective of PETSc]
> >
> > Satish
> >
> 


From balay at mcs.anl.gov  Fri May  8 14:17:36 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 14:17:36 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh4D+yEK4HTr_8qheGQ79g5ajZBWTqdNv+jQRy9jsQ3Ujw@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
	<alpine.LFD.2.11.1505081119570.2737@asterix>
	<CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
	<CADOhEh4D+yEK4HTr_8qheGQ79g5ajZBWTqdNv+jQRy9jsQ3Ujw@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505081412090.2737@asterix>

>>>>>>
Executing: cc -c -o /tmp/petsc-8S0Qxp/config.setCompilers/conftest.o -I/tmp/petsc-8S0Qxp/config.setCompilers   /tmp/petsc-8S0Qxp/config.setCompilers/conftest.c
<snip>
Executing: cc  -o /tmp/petsc-8S0Qxp/config.setCompilers/conftest    /tmp/petsc-8S0Qxp/config.setCompilers/conftest.o -lstdc++
Executing: /tmp/petsc-8S0Qxp/config.setCompilers/conftest
Executing: /tmp/petsc-8S0Qxp/config.setCompilers/conftest
ERROR while running executable: Could not execute "/tmp/petsc-8S0Qxp/config.setCompilers/conftest":
<<<<<<

Hm - there is no error message [or error number] listed.

Perhaps you can try debugging this without PETSc. i.e compile/run a
simple code and see if it works.

You can use src/benchmarks/sizeof.c or
http://trac.mpich.org/projects/mpich/export/60482cc2ca9a740d049444a9261a04a3007bc306/examples/cpi.c

[perhaps runs from $HOME work - but not /tmp ? - but then you said -
'This workflow seemed to work for pgi & gnu programming envs' - so
this shouldn't be the problem]

Satish

On Fri, 8 May 2015, Mark Adams wrote:

> I added the LD path stuff and that seems to have fixed this problem ... but
> there are still problems.
> 
> On Fri, May 8, 2015 at 1:23 PM, David Trebotich <dptrebotich at lbl.gov> wrote:
> 
> > This has all come up because I cannot build Chombo-crunch using gnu
> > prgenv---cannot find mkl anymore.  So olcf consultants advised Intel
> > prgenv.  I am not sure yet what form of libs are used, that is mkl, blas, I
> > don't know.
> >
> > David Trebotich
> > sent from mobile
> > (510) 384-6868
> > On May 8, 2015 9:22 AM, "Satish Balay" <balay at mcs.anl.gov> wrote:
> >
> >> On Fri, 8 May 2015, Ed D'Azevedo wrote:
> >>
> >> > Is petsc using ACML specific routines?
> >> > If not, perhaps MKL provides the lapack/blas and ACML is not needed?
> >> >
> >> > Just a thought.
> >>
> >> yup PETSc can use whatever blas/lapck user provides.
> >>
> >> In this case PETSc was not seeking acml - but the 'module' env was
> >> presumably setup with acml [and the compilers are attempting to use
> >> these modules directly - irrespective of PETSc]
> >>
> >> Satish
> >>
> >
> 


From ganesh.iitm at gmail.com  Fri May  8 16:04:48 2015
From: ganesh.iitm at gmail.com (Ganesh Vijayakumar)
Date: Fri, 8 May 2015 17:04:48 -0400
Subject: [petsc-users] Changing a few entries in a matrix
Message-ID: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>

Hello,

  I have a CFD application where some entries of the matrix change every
time step. However, the total number of non-zero elements never changes. Is
there anyway I can efficiently do this. My current attempts at doing this
results in running over the total initial memory allocation very soon.

ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/297cdbad/attachment.html>

From knepley at gmail.com  Fri May  8 16:10:23 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 16:10:23 -0500
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
Message-ID: <CAMYG4Gnby3zWQ1HXW6sW2mMLpH5BZnLNavdok9M2JYFZdbd0OQ@mail.gmail.com>

On Fri, May 8, 2015 at 4:04 PM, Ganesh Vijayakumar <ganesh.iitm at gmail.com>
wrote:

> Hello,
>
>   I have a CFD application where some entries of the matrix change every
> time step. However, the total number of non-zero elements never changes. Is
> there anyway I can efficiently do this. My current attempts at doing this
> results in running over the total initial memory allocation very soon.
>

If you know where the next elements will be, you can overallocate and fill
those entries with zeros. If not,
then we do not have a good dynamics data structure. This could be done with
hash tables I think, but we
have not had a call for it.

  Thanks,

    Matt



> ganesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/be897f25/attachment.html>

From jed at jedbrown.org  Fri May  8 16:10:32 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 May 2015 17:10:32 -0400
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
Message-ID: <87h9rmbws7.fsf@jedbrown.org>

Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:

> Hello,
>
>   I have a CFD application where some entries of the matrix change every
> time step. 

The entries change value or the set of nonzero entries changes?  If the
former, just MatSetValues and carry on.  If the latter, either allocate
for the union of all possible nonzeros (storing some explicit zeros) or
reallocate the matrix each time.

> However, the total number of non-zero elements never changes. Is there
> anyway I can efficiently do this. My current attempts at doing this
> results in running over the total initial memory allocation very soon.
>
> ganesh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/c09778c3/attachment.pgp>

From ganesh.iitm at gmail.com  Fri May  8 16:14:27 2015
From: ganesh.iitm at gmail.com (Ganesh Vijayakumar)
Date: Fri, 8 May 2015 17:14:27 -0400
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <87h9rmbws7.fsf@jedbrown.org>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
	<87h9rmbws7.fsf@jedbrown.org>
Message-ID: <CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>

Hello,

  Thanks. Looks like I could set all the possible entries I'll use to zero
and selectively overwrite them.

  Can I overwrite any entries without incurring additional cost in memory?
I currently use MatSetValue with the option INSERT_VALUES? Does Petsc have
an option to OVERWRITE the existing value?

ganesh

On Fri, May 8, 2015 at 5:10 PM, Jed Brown <jed at jedbrown.org> wrote:

> Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:
>
> > Hello,
> >
> >   I have a CFD application where some entries of the matrix change every
> > time step.
>
> The entries change value or the set of nonzero entries changes?  If the
> former, just MatSetValues and carry on.  If the latter, either allocate
> for the union of all possible nonzeros (storing some explicit zeros) or
> reallocate the matrix each time.
>
> > However, the total number of non-zero elements never changes. Is there
> > anyway I can efficiently do this. My current attempts at doing this
> > results in running over the total initial memory allocation very soon.
> >
> > ganesh
>



-- 
ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/5a87315c/attachment.html>

From knepley at gmail.com  Fri May  8 16:15:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 8 May 2015 16:15:56 -0500
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
	<87h9rmbws7.fsf@jedbrown.org>
	<CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>
Message-ID: <CAMYG4Gm7KxEjyGjE_8TVCRAGCBM1cJMKzT7otidXfAbMhPGabA@mail.gmail.com>

On Fri, May 8, 2015 at 4:14 PM, Ganesh Vijayakumar <ganesh.iitm at gmail.com>
wrote:

> Hello,
>
>   Thanks. Looks like I could set all the possible entries I'll use to zero
> and selectively overwrite them.
>
>   Can I overwrite any entries without incurring additional cost in memory?
> I currently use MatSetValue with the option INSERT_VALUES? Does Petsc have
> an option to OVERWRITE the existing value?
>

Here INSERT means the same as overwrite. You only incur costs if you have
not previously set the element.

  Thanks,

    Natt


> ganesh
>
> On Fri, May 8, 2015 at 5:10 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:
>>
>> > Hello,
>> >
>> >   I have a CFD application where some entries of the matrix change every
>> > time step.
>>
>> The entries change value or the set of nonzero entries changes?  If the
>> former, just MatSetValues and carry on.  If the latter, either allocate
>> for the union of all possible nonzeros (storing some explicit zeros) or
>> reallocate the matrix each time.
>>
>> > However, the total number of non-zero elements never changes. Is there
>> > anyway I can efficiently do this. My current attempts at doing this
>> > results in running over the total initial memory allocation very soon.
>> >
>> > ganesh
>>
>
>
>
> --
> ganesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/6bef4f23/attachment-0001.html>

From ganesh.iitm at gmail.com  Fri May  8 16:19:23 2015
From: ganesh.iitm at gmail.com (Ganesh Vijayakumar)
Date: Fri, 8 May 2015 17:19:23 -0400
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <CAMYG4Gm7KxEjyGjE_8TVCRAGCBM1cJMKzT7otidXfAbMhPGabA@mail.gmail.com>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
	<87h9rmbws7.fsf@jedbrown.org>
	<CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>
	<CAMYG4Gm7KxEjyGjE_8TVCRAGCBM1cJMKzT7otidXfAbMhPGabA@mail.gmail.com>
Message-ID: <CAFtK+esO=ooUByRLY_HS3y+KQDnYbvpOjpMaRWD1rXv7NuuAdg@mail.gmail.com>

Hello,

On Fri, May 8, 2015 at 5:15 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 8, 2015 at 4:14 PM, Ganesh Vijayakumar <ganesh.iitm at gmail.com>
> wrote:
>
>> Hello,
>>
>>   Thanks. Looks like I could set all the possible entries I'll use to
>> zero and selectively overwrite them.
>>
>>   Can I overwrite any entries without incurring additional cost in
>> memory? I currently use MatSetValue with the option INSERT_VALUES? Does
>> Petsc have an option to OVERWRITE the existing value?
>>
>
> Here INSERT means the same as overwrite. You only incur costs if you have
> not previously set the element.
>

Oh. That was my intial thinking too. Thanks.

ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/808f19bb/attachment.html>

From jed at jedbrown.org  Fri May  8 16:19:31 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 May 2015 17:19:31 -0400
Subject: [petsc-users] Changing a few entries in a matrix
In-Reply-To: <CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>
References: <CAFtK+evfA3K_1jb85jGqC44-G4nH3wOJA0dC-kSTcEJJv2O+cg@mail.gmail.com>
	<87h9rmbws7.fsf@jedbrown.org>
	<CAFtK+evL6Fz6sJecojMf0J4KCeJnBUN_Wh5eun8tKjiE+rY8Zg@mail.gmail.com>
Message-ID: <87bnhubwd8.fsf@jedbrown.org>

Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:
>   Can I overwrite any entries without incurring additional cost in memory?
> I currently use MatSetValue with the option INSERT_VALUES? Does Petsc have
> an option to OVERWRITE the existing value?

INSERT_VALUES is it, in contrast to ADD_VALUES.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/b66f83aa/attachment.pgp>

From ganesh.iitm at gmail.com  Fri May  8 16:21:07 2015
From: ganesh.iitm at gmail.com (Ganesh Vijayakumar)
Date: Fri, 8 May 2015 17:21:07 -0400
Subject: [petsc-users] MatSetValue - Faster if guaranteed local processor?
Message-ID: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>

Hello,

   Is there any way I can get MatSetValue to run faster if I can guarantee
that the entries are local to that processor?

ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/d561877a/attachment.html>

From jed at jedbrown.org  Fri May  8 16:24:43 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 May 2015 17:24:43 -0400
Subject: [petsc-users] MatSetValue - Faster if guaranteed local
	processor?
In-Reply-To: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>
References: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>
Message-ID: <878ucybw4k.fsf@jedbrown.org>

Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:

> Hello,
>
>    Is there any way I can get MatSetValue to run faster if I can guarantee
> that the entries are local to that processor?

Use MatSetValuesBlocked if relevant.  Determining on/off process is not
a bottleneck.  Profile carefully before assuming where the time is
spent.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/724e3249/attachment.pgp>

From ganesh.iitm at gmail.com  Fri May  8 16:30:24 2015
From: ganesh.iitm at gmail.com (Ganesh Vijayakumar)
Date: Fri, 8 May 2015 17:30:24 -0400
Subject: [petsc-users] MatSetValue - Faster if guaranteed local
	processor?
In-Reply-To: <878ucybw4k.fsf@jedbrown.org>
References: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>
	<878ucybw4k.fsf@jedbrown.org>
Message-ID: <CAFtK+esQpVuKLM6Z6VuwNGkmxE86VBChKXKWs-NTZ+C9Pfk10Q@mail.gmail.com>

Hello,

On Fri, May 8, 2015 at 5:24 PM, Jed Brown <jed at jedbrown.org> wrote:

> Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:
>
> > Hello,
> >
> >    Is there any way I can get MatSetValue to run faster if I can
> guarantee
> > that the entries are local to that processor?
>
> Use MatSetValuesBlocked if relevant.  Determining on/off process is not
> a bottleneck.  Profile carefully before assuming where the time is
> spent.
>

Thanks. I have a sparse matrix. The current access order doesn't allow easy
blocking of the values being set.

Thanks for the advice on profiling too. I am working towards it. I was
initially working on getting it working.

ganesh


-- 
ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/4f88cb81/attachment.html>

From balay at mcs.anl.gov  Fri May  8 16:33:41 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 16:33:41 -0500
Subject: [petsc-users] MatSetValue - Faster if guaranteed local
 processor?
In-Reply-To: <878ucybw4k.fsf@jedbrown.org>
References: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>
	<878ucybw4k.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.11.1505081630030.2737@asterix>

On Fri, 8 May 2015, Jed Brown wrote:

> Ganesh Vijayakumar <ganesh.iitm at gmail.com> writes:
> 
> > Hello,
> >
> >    Is there any way I can get MatSetValue to run faster if I can guarantee
> > that the entries are local to that processor?
> 
> Use MatSetValuesBlocked if relevant.  Determining on/off process is not
> a bottleneck.  Profile carefully before assuming where the time is
> spent.

The assembly will be most efficient if:

1. preallocation is done correctly

2. most/all entries are local

3. If you can insert values in blocks [and if the block has multiple
colum values - then the col indices are in sorted accending order]

Satish


From balay at mcs.anl.gov  Fri May  8 16:38:20 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 8 May 2015 16:38:20 -0500
Subject: [petsc-users] MatSetValue - Faster if guaranteed local
 processor?
In-Reply-To: <CAFtK+esQpVuKLM6Z6VuwNGkmxE86VBChKXKWs-NTZ+C9Pfk10Q@mail.gmail.com>
References: <CAFtK+eu07p9k7YsWkK-vDHYhixkDHLea7NsPBbjn71u6naFB8g@mail.gmail.com>
	<878ucybw4k.fsf@jedbrown.org>
	<CAFtK+esQpVuKLM6Z6VuwNGkmxE86VBChKXKWs-NTZ+C9Pfk10Q@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505081635290.2737@asterix>

On Fri, 8 May 2015, Ganesh Vijayakumar wrote:

> I have a sparse matrix. The current access order doesn't allow easy
> blocking of the values being set.

assembling a single row (or part of the row) per MatSetValues() call
would also be a block assembly.

Satish

From amesga1 at tigers.lsu.edu  Fri May  8 16:51:51 2015
From: amesga1 at tigers.lsu.edu (Ataollah Mesgarnejad)
Date: Fri, 8 May 2015 16:51:51 -0500
Subject: [petsc-users] DMPlex and HDF5 vector ordering
In-Reply-To: <CAMYG4Gk5_v1A+PK=ANy0v=4piUEQXLCBgdAOo=-jU+nm66JskQ@mail.gmail.com>
References: <554C5681.7020704@auckland.ac.nz>
	<CAP2=TMitVLADVp9=j5emnHx2AVegi24MG2JXGgf92Jys=C1adQ@mail.gmail.com>
	<CAMYG4Gk5_v1A+PK=ANy0v=4piUEQXLCBgdAOo=-jU+nm66JskQ@mail.gmail.com>
Message-ID: <CAC+VmGfS+eENtRj32b7zNXzFvKmbWftRR69ccLHL+UnO=i6uvw@mail.gmail.com>

Hi,

I just made our fork public at https://bitbucket.org/mesgarnejad/petsc.
It's working progress and nothing is settled yet but you can use it right
now for saving and loading the global Vectors of the DMPlex you are using.

Simply you should first set the global to natural SF by:

  PetscSF G2N;
  ierr =
DMPlexCreateGlobalToNaturalPetscSF(distDM,pointSF,seqSection,&G2N);CHKERRQ(ierr);
  ierr = DMPlexSetGlobalToNaturalPetscSF(distDM,G2N);CHKERRQ(ierr);

where

   -  you get the distDM  and the pointSF from the DMPlexDistribute()
   - seqSection is the data layout for the original DM (I'm trying to fix
   this so you wouldn't need to pass this).


Then when saving and loading  you push native format to your viewer:

  ierr = PetscViewerPushFormat(hdf5Viewer,
 PETSC_VIEWER_NATIVE);CHKERRQ(ierr);


You can see an example for writing and loading the coordinates of a DM over
different number of processors in our fork at
src/dm/impls/plex/examples/tests/ex14.c

Again this working progress so it's subject to changes.

Best,
Ata


On Fri, May 8, 2015 at 7:43 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 8, 2015 at 1:48 AM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> I also had the same issue. My current work around is the following.
>>
>> 1) Run the first DMPlex program on one process and write the vector into
>> HDF5.
>>
>> 2) Run the second DMPlex program with any number of processes but do the
>> following:
>>
>> 3) After you create the initial DMPlex on rank 0, but before distributing
>> it, duplicate it and create its petscsection and vector.
>>
>> 4) Load the HDF5 file into that vector. At this point the ordering is the
>> same.
>>
>> 5) Distribute the original DM and save the PetscSF.
>>
>> 6) Call DMPlexDistributeField() to distribute the vector.
>>
>>
>> This will guarantee the right ordering for the second program no matter
>> how many processes it uses. Only drawback is that the first program has to
>> be run in serial. I am also looking for a better way. Matt any thoughts?
>>
>
> Ata and Blaise have a pull request coming that creates a "natural
> ordering" for a Plex, similar to the
> one used by DMDA, so you get output that is invariant to process number.
> It may take until the end
> of the summer to get it fully integrated, but it is very close.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Justin
>>
>>
>> On Friday, May 8, 2015, Adrian Croucher <a.croucher at auckland.ac.nz>
>> wrote:
>>
>>> hi,
>>>
>>> I create a Vec on a DMPlex using DMPlexCreateGlobalVec(), then write it
>>> to HDF5 using PetscViewerHDF5Open() and VecView().
>>>
>>> I then try to read it back in later (in another program, but using the
>>> same DMPlex) using PetscViewerHDF5Open() and VecLoad().
>>>
>>> It looks like the ordering of the final vector entries in the second
>>> program depends on how many processors I use. If they are the same in both
>>> programs, I get the right ordering, but if they aren't, I don't. Is that
>>> expected? If so, is there any way to guarantee the right ordering when I
>>> read the Vec back in?
>>>
>>> - Adrian
>>>
>>> --
>>> Dr Adrian Croucher
>>> Senior Research Fellow
>>> Department of Engineering Science
>>> University of Auckland, New Zealand
>>> email: a.croucher at auckland.ac.nz
>>> tel: +64 (0)9 923 84611
>>>
>>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
A. Mesgarnejad, Ph.D.
Postdoctoral Researcher
Center for Computation & Technology
Louisiana State University
2093 Digital Media Center,
Baton Rouge, La 70803
www.mesgarnejad.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/512b7cd9/attachment-0001.html>

From jed at jedbrown.org  Fri May  8 19:51:47 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 May 2015 20:51:47 -0400
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <alpine.LFD.2.11.1505081115200.2737@asterix>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
Message-ID: <876182a7z0.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> Also - Perhaps with intel compilers - the recommended blas is
> something other than ACML? [like MKL?]. Something to check..

MKL (and anything built using the Intel compiler) has the "run slower on
AMD" feature.  For any given operation, ACML won't necessarily be
faster, but at least it's not intentionally crippled.  For most of what
PETSc does, any such MKL/ACML difference is irrelevant.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/280abb9c/attachment.pgp>

From jed at jedbrown.org  Fri May  8 22:55:27 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 08 May 2015 23:55:27 -0400
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <CAP2=TMgrw2J9zmYDBdS3VMZ8XQFqvM_EfanLLDporKYzg7oOrg@mail.gmail.com>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org> <87k2wkdyrl.fsf@jedbrown.org>
	<CAMYG4G= 6T=dymx=
	UO=9iVJAG7UNh-_o9=8UU5BfExNVkKzQoHw@mail.gmail.com>
	<87h9rodxwj.fsf@jedbrown.org>
	<CAP2=TMgrw2J9zmYDBdS3VMZ8XQFqvM_EfanLLDporKYzg7oOrg@mail.gmail.com>
Message-ID: <87sib68kwg.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> For clarification purposes:
>
> 1) What is the definition of "performance model" and "cache model"? I see
> those two terms used in this thread but want to know the exact difference
> if any.

Cache model is more specific, in this case telling you what needs to
move from DRAM versus what can be reused.  A cache model is often part
of a performance model (allowing you to estimate bandwidth).  It is
occasionally irrelevant, like with STREAM and sufficiently large sizes,
because there is no opportunity for reuse.

> 2) Is what's described in Dinesh's paper a "cache model"?
> What exactly is the caveat or what are the major assumptions that it makes?

The common first assumption for SpMV is that matrix data is loaded only
once, that vector data is also loaded only once, and that latency to
access vector data is negligible.  These assumptions can be utterly
destroyed if you permute the matrix by a randomized ordering.  The
matrix data will still be accessed only once and contiguously, but (for
large enough matrices) the vector data will almost always generate a
cache miss (high latency) and the vector entry will be evicted before it
is used again.  Moreover, typically nothing else on the cache line will
be used.  For a typical 8-byte scalar and 64-byte cache lines, this
means that for an m?m sparse matrix with N nonzeros, the performance
model predicts

  m*sizeof(int) + N*(sizeof(int) + sizeof(scalar)) + 2*m*sizeof(scalar)

where the first two terms are matrix data and the last are the source
and destination vectors.  In a randomized ordering, you'll need

  m*sizeof(int) + N*(sizeof(int) + sizeof(scalar)) + m*sizeof(scalar) + 8*N*sizeof(scalar)

where the last term now represents that each matrix nonzero triggers a
new cache miss accessing the vector.  This last term clearly swamps
everything else.

Furthermore, in the limit of a small cache, you get very poor temporal
locality from vector entries when using any matrix ordering, so that
last term once again dominates.

> 3) Is quantifying the "useful bandwidth sustained for some level of catch"
> analogous/related to cache register reuse and/or vectorization (e.g., how
> well one can maximize SIMD on the machine if that makes any sense)

SIMD and register reuse are orthogonal to cache reuse and you typically
have less control over them (especially if you don't write assembly).
For irregular sparse operations, even with custom implementations, it
may not be worth the overhead of detecting and arranging vectorizable
steps.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150508/fe49354d/attachment.pgp>

From jychang48 at gmail.com  Sat May  9 06:01:30 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Sat, 9 May 2015 06:01:30 -0500
Subject: [petsc-users] Obtaining bytes per second
In-Reply-To: <87sib68kwg.fsf@jedbrown.org>
References: <CAP2=TMghVOzphh1sJGJDHiXjC6oWwKGWosHeJvkYRU5p7_7k8g@mail.gmail.com>
	<CAP2=TMjeggDPxR-xiRL6rEuaB5+CKduuthdpQ+fQAqb-M0nExA@mail.gmail.com>
	<87bni0jph7.fsf@jedbrown.org>
	<CAP2=TMgV-9k4fhwFZ8GJNenpVnsFxQWjUpZRS6NvEwcOfcwHOg@mail.gmail.com>
	<87twvqgccy.fsf@jedbrown.org>
	<CAMYG4GnZ9-TK9wHdOMcKzPvJ5guziN8-a1Qcj-jwab8_tpg=Hg@mail.gmail.com>
	<87oalxhm1i.fsf@jedbrown.org>
	<CAMYG4GkyMUJg2SzWJuZiCwH+xPEms_A+rnNMDXCVAT-cZ3Wsew@mail.gmail.com>
	<CAP2=TMjopa8n5Tfdf=Wsrw=AYwmRNETkNJo4Gy2XSO7V5YCrcA@mail.gmail.com>
	<87d22dhdqu.fsf@jedbrown.org>
	<CAP2=TMhAj_=6RmnX+kXbh8ky=qnTaU8nzG_7fEtiOqevDSCKCw@mail.gmail.com>
	<CAMYG4GkvqL0RUSmpe1QEXSRWw5iPdazO2oC53YG-m3Din1uxmA@mail.gmail.com>
	<876184fr38.fsf@jedbrown.org>
	<CAMYG4GkNw0MqhrvRzqWccwx6EM7ZD=VaOj9GekYV891D63Stpw@mail.gmail.com>
	<CAP2=TMgzqcHkW2eGp0FxJdL31T0KX0AKJWn9fR3JTRJrVkWu9g@mail.gmail.com>
	<CAMYG4G=VPxAJpuODWzn7H1YEUtSsaJyA1rHuZA1O_-JxNjvpqQ@mail.gmail.com>
	<87mw1gdz3r.fsf@jedbrown.org> <87k2wkdyrl.fsf@jedbrown.org>
	<87h9rodxwj.fsf@jedbrown.org>
	<CAP2=TMgrw2J9zmYDBdS3VMZ8XQFqvM_EfanLLDporKYzg7oOrg@mail.gmail.com>
	<87sib68kwg.fsf@jedbrown.org>
Message-ID: <CAP2=TMjO-_EaY-8Vf=_TGwizADN8-ruZFnZ-GL48=_=Y8djgNw@mail.gmail.com>

Jed thanks that helps a lot. I do have more related questions but they are
not necessarily related to PETSc or the original topic of this thread, so I
shall refer to the computational science stackexchange :)

Thanks,
Justin

On Fri, May 8, 2015 at 10:55 PM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > For clarification purposes:
> >
> > 1) What is the definition of "performance model" and "cache model"? I see
> > those two terms used in this thread but want to know the exact difference
> > if any.
>
> Cache model is more specific, in this case telling you what needs to
> move from DRAM versus what can be reused.  A cache model is often part
> of a performance model (allowing you to estimate bandwidth).  It is
> occasionally irrelevant, like with STREAM and sufficiently large sizes,
> because there is no opportunity for reuse.
>
> > 2) Is what's described in Dinesh's paper a "cache model"?
> > What exactly is the caveat or what are the major assumptions that it
> makes?
>
> The common first assumption for SpMV is that matrix data is loaded only
> once, that vector data is also loaded only once, and that latency to
> access vector data is negligible.  These assumptions can be utterly
> destroyed if you permute the matrix by a randomized ordering.  The
> matrix data will still be accessed only once and contiguously, but (for
> large enough matrices) the vector data will almost always generate a
> cache miss (high latency) and the vector entry will be evicted before it
> is used again.  Moreover, typically nothing else on the cache line will
> be used.  For a typical 8-byte scalar and 64-byte cache lines, this
> means that for an m?m sparse matrix with N nonzeros, the performance
> model predicts
>
>   m*sizeof(int) + N*(sizeof(int) + sizeof(scalar)) + 2*m*sizeof(scalar)
>
> where the first two terms are matrix data and the last are the source
> and destination vectors.  In a randomized ordering, you'll need
>
>   m*sizeof(int) + N*(sizeof(int) + sizeof(scalar)) + m*sizeof(scalar) +
> 8*N*sizeof(scalar)
>
> where the last term now represents that each matrix nonzero triggers a
> new cache miss accessing the vector.  This last term clearly swamps
> everything else.
>
> Furthermore, in the limit of a small cache, you get very poor temporal
> locality from vector entries when using any matrix ordering, so that
> last term once again dominates.
>
> > 3) Is quantifying the "useful bandwidth sustained for some level of
> catch"
> > analogous/related to cache register reuse and/or vectorization (e.g., how
> > well one can maximize SIMD on the machine if that makes any sense)
>
> SIMD and register reuse are orthogonal to cache reuse and you typically
> have less control over them (especially if you don't write assembly).
> For irregular sparse operations, even with custom implementations, it
> may not be worth the overhead of detecting and arranging vectorizable
> steps.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150509/08a88913/attachment.html>

From mfadams at lbl.gov  Sat May  9 14:28:58 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 9 May 2015 15:28:58 -0400
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <alpine.LFD.2.11.1505081412090.2737@asterix>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<554CE15E.90109@ornl.gov>
	<alpine.LFD.2.11.1505081119570.2737@asterix>
	<CAOovb7pkYASQN-B-Cd5EUD=o521f+v+iPZrd-BxTDb7NxPKygA@mail.gmail.com>
	<CADOhEh4D+yEK4HTr_8qheGQ79g5ajZBWTqdNv+jQRy9jsQ3Ujw@mail.gmail.com>
	<alpine.LFD.2.11.1505081412090.2737@asterix>
Message-ID: <CADOhEh6s_FRjBM2FAXsCd_0K8s2epgeSGAT29XZrj+Fb0DjqNg@mail.gmail.com>

On Fri, May 8, 2015 at 3:17 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> >>>>>>
> Executing: cc -c -o /tmp/petsc-8S0Qxp/config.setCompilers/conftest.o
> -I/tmp/petsc-8S0Qxp/config.setCompilers
>  /tmp/petsc-8S0Qxp/config.setCompilers/conftest.c
> <snip>
> Executing: cc  -o /tmp/petsc-8S0Qxp/config.setCompilers/conftest
> /tmp/petsc-8S0Qxp/config.setCompilers/conftest.o -lstdc++
> Executing: /tmp/petsc-8S0Qxp/config.setCompilers/conftest
> Executing: /tmp/petsc-8S0Qxp/config.setCompilers/conftest
> ERROR while running executable: Could not execute
> "/tmp/petsc-8S0Qxp/config.setCompilers/conftest":
> <<<<<<
>
> Hm - there is no error message [or error number] listed.
>
> Perhaps you can try debugging this without PETSc. i.e compile/run a
> simple code and see if it works.
>
> You can use src/benchmarks/sizeof.c or
>
> http://trac.mpich.org/projects/mpich/export/60482cc2ca9a740d049444a9261a04a3007bc306/examples/cpi.c
>
> [perhaps runs from $HOME work - but not /tmp ? - but then you said -
> 'This workflow seemed to work for pgi & gnu programming envs' - so
> this shouldn't be the problem]
>

Yea, I tried this workflow with PGI and it worked.  I'm giving up on Intel
and see if we can make GNU work for Treb.

Thanks,


>
> Satish
>
> On Fri, 8 May 2015, Mark Adams wrote:
>
> > I added the LD path stuff and that seems to have fixed this problem ...
> but
> > there are still problems.
> >
> > On Fri, May 8, 2015 at 1:23 PM, David Trebotich <dptrebotich at lbl.gov>
> wrote:
> >
> > > This has all come up because I cannot build Chombo-crunch using gnu
> > > prgenv---cannot find mkl anymore.  So olcf consultants advised Intel
> > > prgenv.  I am not sure yet what form of libs are used, that is mkl,
> blas, I
> > > don't know.
> > >
> > > David Trebotich
> > > sent from mobile
> > > (510) 384-6868
> > > On May 8, 2015 9:22 AM, "Satish Balay" <balay at mcs.anl.gov> wrote:
> > >
> > >> On Fri, 8 May 2015, Ed D'Azevedo wrote:
> > >>
> > >> > Is petsc using ACML specific routines?
> > >> > If not, perhaps MKL provides the lapack/blas and ACML is not needed?
> > >> >
> > >> > Just a thought.
> > >>
> > >> yup PETSc can use whatever blas/lapck user provides.
> > >>
> > >> In this case PETSc was not seeking acml - but the 'module' env was
> > >> presumably setup with acml [and the compilers are attempting to use
> > >> these modules directly - irrespective of PETSc]
> > >>
> > >> Satish
> > >>
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150509/d012b3fc/attachment.html>

From gideon.simpson at gmail.com  Sun May 10 11:26:55 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Sun, 10 May 2015 12:26:55 -0400
Subject: [petsc-users] TS without an RHSJacobian
Message-ID: <A07C571C-6D97-4E8F-A674-CA56FA960A93@gmail.com>

What happens if I use TS for a nonlinear problem without specifying the RHSJacobian?  There?s some mention in the manual that if you?re working with a problem built on a da, it will use finite differences to approximate it, but I found the language a bit ambiguous.  Could someone elaborate? 
-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150510/7085e793/attachment.html>

From knepley at gmail.com  Sun May 10 11:29:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 10 May 2015 11:29:35 -0500
Subject: [petsc-users] TS without an RHSJacobian
In-Reply-To: <A07C571C-6D97-4E8F-A674-CA56FA960A93@gmail.com>
References: <A07C571C-6D97-4E8F-A674-CA56FA960A93@gmail.com>
Message-ID: <CAMYG4Gk+JyWxPFrse7b37hoP11WN542VraeqLfwmOX91ZMDpMg@mail.gmail.com>

On Sun, May 10, 2015 at 11:26 AM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> What happens if I use TS for a nonlinear problem without specifying the
> RHSJacobian?  There?s some mention in the manual that if you?re working
> with a problem built on a da, it will use finite differences to approximate
> it, but I found the language a bit ambiguous.  Could someone elaborate?
>

It should use finite differences and coloring to make the Jacobian.
However, I have not checked whether its actually implemented.
If not, we will do it. I know I put it into IJacobian().

   Matt


>
> -gideon
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150510/f514876a/attachment.html>

From bikash at umich.edu  Sun May 10 16:33:49 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Sun, 10 May 2015 17:33:49 -0400
Subject: [petsc-users] MatCreateNest
Message-ID: <CA+H+vqJWA=+FMwoO-zz-_s6KRf65SLB5zhEQv6mxVuZXydkX5Q@mail.gmail.com>

Hi,

I'm using MatCreateNest where I assemble four submatrices: A00, A01, A10
and A11 into matrix A. For me A00, A01 and A10 are parallel matrices. But
A11 is a sequential matrix present in all processors. Is there any
restriction on the type of the submatrices used in MatCreateNest , i.e.,
that all must be parallel or sequential but not a mix of parallel and
sequential matrices?

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150510/344fc5c9/attachment.html>

From knepley at gmail.com  Sun May 10 17:05:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 10 May 2015 17:05:16 -0500
Subject: [petsc-users] MatCreateNest
In-Reply-To: <CA+H+vqJWA=+FMwoO-zz-_s6KRf65SLB5zhEQv6mxVuZXydkX5Q@mail.gmail.com>
References: <CA+H+vqJWA=+FMwoO-zz-_s6KRf65SLB5zhEQv6mxVuZXydkX5Q@mail.gmail.com>
Message-ID: <CAMYG4G=BEroDWnR+TPyiRFv6At6oO04TdpBDJQfpJ3bSuC+axA@mail.gmail.com>

On Sun, May 10, 2015 at 4:33 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I'm using MatCreateNest where I assemble four submatrices: A00, A01, A10
> and A11 into matrix A. For me A00, A01 and A10 are parallel matrices. But
> A11 is a sequential matrix present in all processors. Is there any
> restriction on the type of the submatrices used in MatCreateNest , i.e.,
> that all must be parallel or sequential but not a mix of parallel and
> sequential matrices?
>

I believe all matrices need to have the same comm, so you would need to
either

  a) make an MPIAIJ matrix which only has rows on some process, or

  b) divide the rows among processes

   Matt


> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150510/abbbe0d9/attachment.html>

From a.croucher at auckland.ac.nz  Sun May 10 18:31:10 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Mon, 11 May 2015 11:31:10 +1200
Subject: [petsc-users] DMPlex and HDF5 vector ordering
In-Reply-To: <CAMYG4Gk5_v1A+PK=ANy0v=4piUEQXLCBgdAOo=-jU+nm66JskQ@mail.gmail.com>
References: <554C5681.7020704@auckland.ac.nz>	<CAP2=TMitVLADVp9=j5emnHx2AVegi24MG2JXGgf92Jys=C1adQ@mail.gmail.com>
	<CAMYG4Gk5_v1A+PK=ANy0v=4piUEQXLCBgdAOo=-jU+nm66JskQ@mail.gmail.com>
Message-ID: <554FEA3E.9090005@auckland.ac.nz>



On 09/05/15 00:43, Matthew Knepley wrote:
>
> Ata and Blaise have a pull request coming that creates a "natural 
> ordering" for a Plex, similar to the
> one used by DMDA, so you get output that is invariant to process 
> number. It may take until the end
> of the summer to get it fully integrated, but it is very close.

Great, that sounds just like what I need. I'll wait until I see it 
merged into next branch before giving it a go (not totally urgent right 
now).

Thanks!

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/e63f4e97/attachment.html>

From bikash at umich.edu  Mon May 11 07:44:13 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 08:44:13 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
Message-ID: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>

Hi,

I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there any
way to perform A*B after B has been assembled from its sub-matrices?

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/bffede52/attachment.html>

From patrick.sanan at gmail.com  Mon May 11 08:21:45 2015
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Mon, 11 May 2015 15:21:45 +0200
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
Message-ID: <CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>

MatMatMult with MATNEST does not seem to be supported, based only on the
fact that there are no functions of the form MatMatMult_*_MatNest defined
with the MATNEST implementation :
http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html

Does this operation need to be efficient? (That is, are you forming this
matrix for debugging or experimental purposes, or with the intention of
using it within an efficient, scalable piece of code?). If not, it should
be possible to use lower-level matrix operations as defined by the API to
extract the appropriately-sized submatrices from A and (assuming that
MatMatMult is defined between MATMPIAIJ and the submatrices of your
Matnest), perform the matrix multiplications and additions "by hand" .

On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there any
> way to perform A*B after B has been assembled from its sub-matrices?
>
> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/796b0749/attachment.html>

From bikash at umich.edu  Mon May 11 08:47:49 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 09:47:49 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
Message-ID: <CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>

Hi Patrick,

Thanks for the clarification. I want it to be an efficient operation. The
idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x) to a
standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST whereas
H is MPIAIJ. In my problem M^{-1} is computed once whereas H changes every
iteration. So performing low-level matrix multiplication or creating H as
MATNEST and assembling it from its sub-matrices at every iteration sounds
inefficient.

Regards,
Bikash

On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com>
wrote:

> MatMatMult with MATNEST does not seem to be supported, based only on the
> fact that there are no functions of the form MatMatMult_*_MatNest defined
> with the MATNEST implementation :
> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>
> Does this operation need to be efficient? (That is, are you forming this
> matrix for debugging or experimental purposes, or with the intention of
> using it within an efficient, scalable piece of code?). If not, it should
> be possible to use lower-level matrix operations as defined by the API to
> extract the appropriately-sized submatrices from A and (assuming that
> MatMatMult is defined between MATMPIAIJ and the submatrices of your
> Matnest), perform the matrix multiplications and additions "by hand" .
>
> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi,
>>
>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there any
>> way to perform A*B after B has been assembled from its sub-matrices?
>>
>> Thanks,
>> Bikash
>>
>> --
>> Bikash S. Kanungo
>> PhD Student
>> Computational Materials Physics Group
>> Mechanical Engineering
>> University of Michigan
>>
>>
>


-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/3c4fb0cf/attachment-0001.html>

From bikash at umich.edu  Mon May 11 09:18:44 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 10:18:44 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+z91Tfes=O-SPKyvMRRY0QFF0_UmrPSw4OOmitpN+0ks4UsAA@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CA+z91Tfes=O-SPKyvMRRY0QFF0_UmrPSw4OOmitpN+0ks4UsAA@mail.gmail.com>
Message-ID: <CA+H+vqJ=Bi4DmQHr26pOk42VMoJzo1sKCzxNuKfOiJvUvV7=Pw@mail.gmail.com>

Thanks Patrick. I'll give it a shot and see if it works.

Regards,
Bikash

On Mon, May 11, 2015 at 9:52 AM, Patrick Sanan <patrick.sanan at gmail.com>
wrote:

> Ah, so I misunderstood the question - you want to *apply* AB, not assemble
> it. Perhaps you can then use MATCOMPOSITE to define a composite operator
> which applies AB.
>
> On Mon, May 11, 2015 at 3:47 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi Patrick,
>>
>> Thanks for the clarification. I want it to be an efficient operation. The
>> idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x) to a
>> standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST whereas
>> H is MPIAIJ. In my problem M^{-1} is computed once whereas H changes every
>> iteration. So performing low-level matrix multiplication or creating H as
>> MATNEST and assembling it from its sub-matrices at every iteration sounds
>> inefficient.
>>
>> Regards,
>> Bikash
>>
>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com>
>> wrote:
>>
>>> MatMatMult with MATNEST does not seem to be supported, based only on the
>>> fact that there are no functions of the form MatMatMult_*_MatNest defined
>>> with the MATNEST implementation :
>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>
>>> Does this operation need to be efficient? (That is, are you forming this
>>> matrix for debugging or experimental purposes, or with the intention of
>>> using it within an efficient, scalable piece of code?). If not, it should
>>> be possible to use lower-level matrix operations as defined by the API to
>>> extract the appropriately-sized submatrices from A and (assuming that
>>> MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>
>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there
>>>> any way to perform A*B after B has been assembled from its sub-matrices?
>>>>
>>>> Thanks,
>>>> Bikash
>>>>
>>>> --
>>>> Bikash S. Kanungo
>>>> PhD Student
>>>> Computational Materials Physics Group
>>>> Mechanical Engineering
>>>> University of Michigan
>>>>
>>>>
>>>
>>
>>
>> --
>> Bikash S. Kanungo
>> PhD Student
>> Computational Materials Physics Group
>> Mechanical Engineering
>> University of Michigan
>>
>>
>


-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/a555ba52/attachment.html>

From popov at uni-mainz.de  Mon May 11 11:43:46 2015
From: popov at uni-mainz.de (anton)
Date: Mon, 11 May 2015 18:43:46 +0200
Subject: [petsc-users] MatSetValuesStencil
Message-ID: <5550DC42.9010206@uni-mainz.de>

Hi,

MatSetValuesStencil page doesn't contain the standard phrase anymore: 
"Negative indices may be passed in idxm and idxn, these rows and columns 
are simply ignored" . Instead it specifies the usage of negative 
indicies for setting periodic BC.

Can I still use negative indices with MatSetValuesStencil in the usual 
way if I don't use DM_BOUNDARY_PERIODIC boundary type?

Thanks,

Anton



From knepley at gmail.com  Mon May 11 12:00:08 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 11 May 2015 12:00:08 -0500
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
Message-ID: <CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>

On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi Patrick,
>
> Thanks for the clarification. I want it to be an efficient operation. The
> idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x) to a
> standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST whereas
> H is MPIAIJ. In my problem M^{-1} is computed once whereas H changes every
> iteration. So performing low-level matrix multiplication or creating H as
> MATNEST and assembling it from its sub-matrices at every iteration sounds
> inefficient.
>

You can solve these types of things with SLEPc without explicit MatMat.

   Matt


> Regards,
> Bikash
>
> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com>
> wrote:
>
>> MatMatMult with MATNEST does not seem to be supported, based only on the
>> fact that there are no functions of the form MatMatMult_*_MatNest defined
>> with the MATNEST implementation :
>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>
>> Does this operation need to be efficient? (That is, are you forming this
>> matrix for debugging or experimental purposes, or with the intention of
>> using it within an efficient, scalable piece of code?). If not, it should
>> be possible to use lower-level matrix operations as defined by the API to
>> extract the appropriately-sized submatrices from A and (assuming that
>> MatMatMult is defined between MATMPIAIJ and the submatrices of your
>> Matnest), perform the matrix multiplications and additions "by hand" .
>>
>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>>
>>> Hi,
>>>
>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there
>>> any way to perform A*B after B has been assembled from its sub-matrices?
>>>
>>> Thanks,
>>> Bikash
>>>
>>> --
>>> Bikash S. Kanungo
>>> PhD Student
>>> Computational Materials Physics Group
>>> Mechanical Engineering
>>> University of Michigan
>>>
>>>
>>
>
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/65c363d7/attachment.html>

From bikash at umich.edu  Mon May 11 12:12:06 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 13:12:06 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
Message-ID: <CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>

Hi Matthew,

Yes SLEPc does allow me to solve a generalized eigenvalue problem. But the
generalized solvers are not as robust as the standard eigenvalue solvers.
That's the reason I want to use explicit MatMatMult so that I can use
standard eigenvalue solvers.

Thanks,
Bikash
On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:

> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi Patrick,
>>
>> Thanks for the clarification. I want it to be an efficient operation. The
>> idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x) to a
>> standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST whereas
>> H is MPIAIJ. In my problem M^{-1} is computed once whereas H changes every
>> iteration. So performing low-level matrix multiplication or creating H as
>> MATNEST and assembling it from its sub-matrices at every iteration sounds
>> inefficient.
>>
>
> You can solve these types of things with SLEPc without explicit MatMat.
>
>    Matt
>
>
>> Regards,
>> Bikash
>>
>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com>
>> wrote:
>>
>>> MatMatMult with MATNEST does not seem to be supported, based only on the
>>> fact that there are no functions of the form MatMatMult_*_MatNest defined
>>> with the MATNEST implementation :
>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>
>>> Does this operation need to be efficient? (That is, are you forming this
>>> matrix for debugging or experimental purposes, or with the intention of
>>> using it within an efficient, scalable piece of code?). If not, it should
>>> be possible to use lower-level matrix operations as defined by the API to
>>> extract the appropriately-sized submatrices from A and (assuming that
>>> MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>
>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there
>>>> any way to perform A*B after B has been assembled from its sub-matrices?
>>>>
>>>> Thanks,
>>>> Bikash
>>>>
>>>> --
>>>> Bikash S. Kanungo
>>>> PhD Student
>>>> Computational Materials Physics Group
>>>> Mechanical Engineering
>>>> University of Michigan
>>>>
>>>>
>>>
>>
>>
>> --
>> Bikash S. Kanungo
>> PhD Student
>> Computational Materials Physics Group
>> Mechanical Engineering
>> University of Michigan
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/31bdf049/attachment.html>

From patrick.sanan at gmail.com  Mon May 11 12:29:43 2015
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Mon, 11 May 2015 19:29:43 +0200
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
Message-ID: <CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>

Are you storing M^{-1} explicitly? In that case you could use MATCOMPOSITE
to define M^{-1}H and feed that to SLEPc's standard solver. If you only
have M explicitly, you could use MATSHELL to define your own operator which
applies M^{-1}H and use that with SLEPc's standard solver (though of course
SLEPc would be free to implement its generalized eigensolvers this way
and/or in better ways for certain problems - I am not enough of an expert
on that library to say more).

On Mon, May 11, 2015 at 7:12 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi Matthew,
>
> Yes SLEPc does allow me to solve a generalized eigenvalue problem. But the
> generalized solvers are not as robust as the standard eigenvalue solvers.
> That's the reason I want to use explicit MatMatMult so that I can use
> standard eigenvalue solvers.
>
> Thanks,
> Bikash
> On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>
>> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu> wrote:
>>
>>> Hi Patrick,
>>>
>>> Thanks for the clarification. I want it to be an efficient operation.
>>> The idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x)
>>> to a standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST
>>> whereas H is MPIAIJ. In my problem M^{-1} is computed once whereas H
>>> changes every iteration. So performing low-level matrix multiplication or
>>> creating H as MATNEST and assembling it from its sub-matrices at every
>>> iteration sounds inefficient.
>>>
>>
>> You can solve these types of things with SLEPc without explicit MatMat.
>>
>>    Matt
>>
>>
>>> Regards,
>>> Bikash
>>>
>>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com>
>>> wrote:
>>>
>>>> MatMatMult with MATNEST does not seem to be supported, based only on
>>>> the fact that there are no functions of the form MatMatMult_*_MatNest
>>>> defined with the MATNEST implementation :
>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>>
>>>> Does this operation need to be efficient? (That is, are you forming
>>>> this matrix for debugging or experimental purposes, or with the intention
>>>> of using it within an efficient, scalable piece of code?). If not, it
>>>> should be possible to use lower-level matrix operations as defined by the
>>>> API to extract the appropriately-sized submatrices from A and (assuming
>>>> that MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>>
>>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there
>>>>> any way to perform A*B after B has been assembled from its sub-matrices?
>>>>>
>>>>> Thanks,
>>>>> Bikash
>>>>>
>>>>> --
>>>>> Bikash S. Kanungo
>>>>> PhD Student
>>>>> Computational Materials Physics Group
>>>>> Mechanical Engineering
>>>>> University of Michigan
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Bikash S. Kanungo
>>> PhD Student
>>> Computational Materials Physics Group
>>> Mechanical Engineering
>>> University of Michigan
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/6c2da707/attachment.html>

From bikash at umich.edu  Mon May 11 12:37:12 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 13:37:12 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
Message-ID: <CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>

Hi Patrick,

I have M^{-1} computed explicitly and stored as MATNEST. Providing
MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
option. Does it has any additional cost of explicitly building
On May 11, 2015 1:30 PM, "Patrick Sanan" <patrick.sanan at gmail.com> wrote:

> Are you storing M^{-1} explicitly? In that case you could use MATCOMPOSITE
> to define M^{-1}H and feed that to SLEPc's standard solver. If you only
> have M explicitly, you could use MATSHELL to define your own operator which
> applies M^{-1}H and use that with SLEPc's standard solver (though of course
> SLEPc would be free to implement its generalized eigensolvers this way
> and/or in better ways for certain problems - I am not enough of an expert
> on that library to say more).
>
> On Mon, May 11, 2015 at 7:12 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi Matthew,
>>
>> Yes SLEPc does allow me to solve a generalized eigenvalue problem. But
>> the generalized solvers are not as robust as the standard eigenvalue
>> solvers. That's the reason I want to use explicit MatMatMult so that I can
>> use standard eigenvalue solvers.
>>
>> Thanks,
>> Bikash
>> On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>>
>>> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu>
>>> wrote:
>>>
>>>> Hi Patrick,
>>>>
>>>> Thanks for the clarification. I want it to be an efficient operation.
>>>> The idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x)
>>>> to a standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST
>>>> whereas H is MPIAIJ. In my problem M^{-1} is computed once whereas H
>>>> changes every iteration. So performing low-level matrix multiplication or
>>>> creating H as MATNEST and assembling it from its sub-matrices at every
>>>> iteration sounds inefficient.
>>>>
>>>
>>> You can solve these types of things with SLEPc without explicit MatMat.
>>>
>>>    Matt
>>>
>>>
>>>> Regards,
>>>> Bikash
>>>>
>>>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <patrick.sanan at gmail.com
>>>> > wrote:
>>>>
>>>>> MatMatMult with MATNEST does not seem to be supported, based only on
>>>>> the fact that there are no functions of the form MatMatMult_*_MatNest
>>>>> defined with the MATNEST implementation :
>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>>>
>>>>> Does this operation need to be efficient? (That is, are you forming
>>>>> this matrix for debugging or experimental purposes, or with the intention
>>>>> of using it within an efficient, scalable piece of code?). If not, it
>>>>> should be possible to use lower-level matrix operations as defined by the
>>>>> API to extract the appropriately-sized submatrices from A and (assuming
>>>>> that MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>>>
>>>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is there
>>>>>> any way to perform A*B after B has been assembled from its sub-matrices?
>>>>>>
>>>>>> Thanks,
>>>>>> Bikash
>>>>>>
>>>>>> --
>>>>>> Bikash S. Kanungo
>>>>>> PhD Student
>>>>>> Computational Materials Physics Group
>>>>>> Mechanical Engineering
>>>>>> University of Michigan
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Bikash S. Kanungo
>>>> PhD Student
>>>> Computational Materials Physics Group
>>>> Mechanical Engineering
>>>> University of Michigan
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/426e482f/attachment-0001.html>

From bikash at umich.edu  Mon May 11 12:41:12 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 13:41:12 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
Message-ID: <CA+H+vqKVfF02nV+D5+-2FrPrMuUmn+p_Zu-EGWoRNHaVqUwmKA@mail.gmail.com>

Hi Patrick,

I have M^{-1} computed explicitly and stored as MATNEST. Providing
MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
option. Does it has any additional cost over explicitly building M^{-1}H
(assuming I have a way to do so) and then providing it to standard
eigenvalue solvers?

Thanks,

Bikash

On Mon, May 11, 2015 at 1:37 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi Patrick,
>
> I have M^{-1} computed explicitly and stored as MATNEST. Providing
> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
> option. Does it has any additional cost of explicitly building
> On May 11, 2015 1:30 PM, "Patrick Sanan" <patrick.sanan at gmail.com> wrote:
>
>> Are you storing M^{-1} explicitly? In that case you could use
>> MATCOMPOSITE to define M^{-1}H and feed that to SLEPc's standard solver. If
>> you only have M explicitly, you could use MATSHELL to define your own
>> operator which applies M^{-1}H and use that with SLEPc's standard solver
>> (though of course SLEPc would be free to implement its generalized
>> eigensolvers this way and/or in better ways for certain problems - I am not
>> enough of an expert on that library to say more).
>>
>> On Mon, May 11, 2015 at 7:12 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>>
>>> Hi Matthew,
>>>
>>> Yes SLEPc does allow me to solve a generalized eigenvalue problem. But
>>> the generalized solvers are not as robust as the standard eigenvalue
>>> solvers. That's the reason I want to use explicit MatMatMult so that I can
>>> use standard eigenvalue solvers.
>>>
>>> Thanks,
>>> Bikash
>>> On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>>>
>>>> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu>
>>>> wrote:
>>>>
>>>>> Hi Patrick,
>>>>>
>>>>> Thanks for the clarification. I want it to be an efficient operation.
>>>>> The idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x)
>>>>> to a standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST
>>>>> whereas H is MPIAIJ. In my problem M^{-1} is computed once whereas H
>>>>> changes every iteration. So performing low-level matrix multiplication or
>>>>> creating H as MATNEST and assembling it from its sub-matrices at every
>>>>> iteration sounds inefficient.
>>>>>
>>>>
>>>> You can solve these types of things with SLEPc without explicit MatMat.
>>>>
>>>>    Matt
>>>>
>>>>
>>>>> Regards,
>>>>> Bikash
>>>>>
>>>>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <
>>>>> patrick.sanan at gmail.com> wrote:
>>>>>
>>>>>> MatMatMult with MATNEST does not seem to be supported, based only on
>>>>>> the fact that there are no functions of the form MatMatMult_*_MatNest
>>>>>> defined with the MATNEST implementation :
>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>>>>
>>>>>> Does this operation need to be efficient? (That is, are you forming
>>>>>> this matrix for debugging or experimental purposes, or with the intention
>>>>>> of using it within an efficient, scalable piece of code?). If not, it
>>>>>> should be possible to use lower-level matrix operations as defined by the
>>>>>> API to extract the appropriately-sized submatrices from A and (assuming
>>>>>> that MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>>>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>>>>
>>>>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is
>>>>>>> there any way to perform A*B after B has been assembled from its
>>>>>>> sub-matrices?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bikash
>>>>>>>
>>>>>>> --
>>>>>>> Bikash S. Kanungo
>>>>>>> PhD Student
>>>>>>> Computational Materials Physics Group
>>>>>>> Mechanical Engineering
>>>>>>> University of Michigan
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bikash S. Kanungo
>>>>> PhD Student
>>>>> Computational Materials Physics Group
>>>>> Mechanical Engineering
>>>>> University of Michigan
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>


-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/738dfa53/attachment.html>

From jed at jedbrown.org  Mon May 11 12:41:46 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 May 2015 13:41:46 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
Message-ID: <87r3qn57vp.fsf@jedbrown.org>

Bikash Kanungo <bikash at umich.edu> writes:

> Hi Patrick,
>
> I have M^{-1} computed explicitly and stored as MATNEST. Providing
> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
> option. Does it has any additional cost of explicitly building

Why are you storing M^{-1} as MATNEST?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/ca121ece/attachment.pgp>

From knepley at gmail.com  Mon May 11 12:42:59 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 11 May 2015 12:42:59 -0500
Subject: [petsc-users] MatSetValuesStencil
In-Reply-To: <5550DC42.9010206@uni-mainz.de>
References: <5550DC42.9010206@uni-mainz.de>
Message-ID: <CAMYG4GmU9WX8gRdTQZSNVO30JW8YzCr-oXZb_tDGhnzRN3te7Q@mail.gmail.com>

On Mon, May 11, 2015 at 11:43 AM, anton <popov at uni-mainz.de> wrote:

> Hi,
>
> MatSetValuesStencil page doesn't contain the standard phrase anymore:
> "Negative indices may be passed in idxm and idxn, these rows and columns
> are simply ignored" . Instead it specifies the usage of negative indicies
> for setting periodic BC.
>
> Can I still use negative indices with MatSetValuesStencil in the usual way
> if I don't use DM_BOUNDARY_PERIODIC boundary type?
>

Yes.

  Thanks,

    Matt


> Thanks,
>
> Anton
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/6993c031/attachment.html>

From bikash at umich.edu  Mon May 11 12:47:10 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 13:47:10 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <87r3qn57vp.fsf@jedbrown.org>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<87r3qn57vp.fsf@jedbrown.org>
Message-ID: <CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>

Hi Jed,

The physics of the problem requires evaluation of M^{-1} from four
sub-matrices. That's why I'm constrained to use it as MATNEST. The other
option seems to be MatGetLocalSubMatrix but it seemed easier for me to go
with MATNEST (particularly to avoid the local index sets).

Regards,
Bikash
On May 11, 2015 1:41 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Bikash Kanungo <bikash at umich.edu> writes:
>
> > Hi Patrick,
> >
> > I have M^{-1} computed explicitly and stored as MATNEST. Providing
> > MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
> > option. Does it has any additional cost of explicitly building
>
> Why are you storing M^{-1} as MATNEST?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/09319c67/attachment.html>

From bikash at umich.edu  Mon May 11 12:51:04 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 13:51:04 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+z91Tf-DuhT8UEn4ruYFuVX2z8pzrxD+5PbbWMPfAK4z+AiUA@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<CA+H+vqKVfF02nV+D5+-2FrPrMuUmn+p_Zu-EGWoRNHaVqUwmKA@mail.gmail.com>
	<CA+z91Tf-DuhT8UEn4ruYFuVX2z8pzrxD+5PbbWMPfAK4z+AiUA@mail.gmail.com>
Message-ID: <CA+H+vq+y_1oZVFBoZcBKY-dBWfa=RKpTU6fSo6yhWKBrJ7c6Cw@mail.gmail.com>

Cool. I guess I'll have to profile both ways: MATCOMPOSITE and explicit
MatMat.
On May 11, 2015 1:47 PM, "Patrick Sanan" <patrick.sanan at gmail.com> wrote:

> I think that depends on how sparse M^{-1} and H are.
>
> On Mon, May 11, 2015 at 7:41 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi Patrick,
>>
>> I have M^{-1} computed explicitly and stored as MATNEST. Providing
>> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
>> option. Does it has any additional cost over explicitly building M^{-1}H
>> (assuming I have a way to do so) and then providing it to standard
>> eigenvalue solvers?
>>
>> Thanks,
>>
>> Bikash
>>
>> On Mon, May 11, 2015 at 1:37 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>>
>>> Hi Patrick,
>>>
>>> I have M^{-1} computed explicitly and stored as MATNEST. Providing
>>> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
>>> option. Does it has any additional cost of explicitly building
>>> On May 11, 2015 1:30 PM, "Patrick Sanan" <patrick.sanan at gmail.com>
>>> wrote:
>>>
>>>> Are you storing M^{-1} explicitly? In that case you could use
>>>> MATCOMPOSITE to define M^{-1}H and feed that to SLEPc's standard solver. If
>>>> you only have M explicitly, you could use MATSHELL to define your own
>>>> operator which applies M^{-1}H and use that with SLEPc's standard solver
>>>> (though of course SLEPc would be free to implement its generalized
>>>> eigensolvers this way and/or in better ways for certain problems - I am not
>>>> enough of an expert on that library to say more).
>>>>
>>>> On Mon, May 11, 2015 at 7:12 PM, Bikash Kanungo <bikash at umich.edu>
>>>> wrote:
>>>>
>>>>> Hi Matthew,
>>>>>
>>>>> Yes SLEPc does allow me to solve a generalized eigenvalue problem. But
>>>>> the generalized solvers are not as robust as the standard eigenvalue
>>>>> solvers. That's the reason I want to use explicit MatMatMult so that I can
>>>>> use standard eigenvalue solvers.
>>>>>
>>>>> Thanks,
>>>>> Bikash
>>>>> On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>>>>>
>>>>>> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Patrick,
>>>>>>>
>>>>>>> Thanks for the clarification. I want it to be an efficient
>>>>>>> operation. The idea is to convert a generalized eigenvalue problem (H*x =
>>>>>>> lamda*M*x) to a standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type
>>>>>>> MATNEST whereas H is MPIAIJ. In my problem M^{-1} is computed once whereas
>>>>>>> H changes every iteration. So performing low-level matrix multiplication or
>>>>>>> creating H as MATNEST and assembling it from its sub-matrices at every
>>>>>>> iteration sounds inefficient.
>>>>>>>
>>>>>>
>>>>>> You can solve these types of things with SLEPc without explicit
>>>>>> MatMat.
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>
>>>>>>> Regards,
>>>>>>> Bikash
>>>>>>>
>>>>>>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <
>>>>>>> patrick.sanan at gmail.com> wrote:
>>>>>>>
>>>>>>>> MatMatMult with MATNEST does not seem to be supported, based only
>>>>>>>> on the fact that there are no functions of the form MatMatMult_*_MatNest
>>>>>>>> defined with the MATNEST implementation :
>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>>>>>>
>>>>>>>> Does this operation need to be efficient? (That is, are you forming
>>>>>>>> this matrix for debugging or experimental purposes, or with the intention
>>>>>>>> of using it within an efficient, scalable piece of code?). If not, it
>>>>>>>> should be possible to use lower-level matrix operations as defined by the
>>>>>>>> API to extract the appropriately-sized submatrices from A and (assuming
>>>>>>>> that MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>>>>>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>>>>>>
>>>>>>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is
>>>>>>>>> there any way to perform A*B after B has been assembled from its
>>>>>>>>> sub-matrices?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bikash
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bikash S. Kanungo
>>>>>>>>> PhD Student
>>>>>>>>> Computational Materials Physics Group
>>>>>>>>> Mechanical Engineering
>>>>>>>>> University of Michigan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Bikash S. Kanungo
>>>>>>> PhD Student
>>>>>>> Computational Materials Physics Group
>>>>>>> Mechanical Engineering
>>>>>>> University of Michigan
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>
>>
>>
>> --
>> Bikash S. Kanungo
>> PhD Student
>> Computational Materials Physics Group
>> Mechanical Engineering
>> University of Michigan
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/676ae57d/attachment-0001.html>

From patrick.sanan at gmail.com  Mon May 11 12:51:17 2015
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Mon, 11 May 2015 19:51:17 +0200
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqKVfF02nV+D5+-2FrPrMuUmn+p_Zu-EGWoRNHaVqUwmKA@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<CA+H+vqKVfF02nV+D5+-2FrPrMuUmn+p_Zu-EGWoRNHaVqUwmKA@mail.gmail.com>
Message-ID: <CA+z91Tf+Vg5sgPXb2oBs_FuU+GPrPydFaVJX7N1WJUdBUPA6Lw@mail.gmail.com>

(forgot to reply-all)
I think that depends on how sparse M^{-1} and H are.



On Mon, May 11, 2015 at 7:41 PM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi Patrick,
>
> I have M^{-1} computed explicitly and stored as MATNEST. Providing
> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
> option. Does it has any additional cost over explicitly building M^{-1}H
> (assuming I have a way to do so) and then providing it to standard
> eigenvalue solvers?
>
> Thanks,
>
> Bikash
>
> On Mon, May 11, 2015 at 1:37 PM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi Patrick,
>>
>> I have M^{-1} computed explicitly and stored as MATNEST. Providing
>> MATCOMPOSITE of M^{-1}H to SLEPc standard eigenvalue solvers seems an
>> option. Does it has any additional cost of explicitly building
>> On May 11, 2015 1:30 PM, "Patrick Sanan" <patrick.sanan at gmail.com> wrote:
>>
>>> Are you storing M^{-1} explicitly? In that case you could use
>>> MATCOMPOSITE to define M^{-1}H and feed that to SLEPc's standard solver. If
>>> you only have M explicitly, you could use MATSHELL to define your own
>>> operator which applies M^{-1}H and use that with SLEPc's standard solver
>>> (though of course SLEPc would be free to implement its generalized
>>> eigensolvers this way and/or in better ways for certain problems - I am not
>>> enough of an expert on that library to say more).
>>>
>>> On Mon, May 11, 2015 at 7:12 PM, Bikash Kanungo <bikash at umich.edu>
>>> wrote:
>>>
>>>> Hi Matthew,
>>>>
>>>> Yes SLEPc does allow me to solve a generalized eigenvalue problem. But
>>>> the generalized solvers are not as robust as the standard eigenvalue
>>>> solvers. That's the reason I want to use explicit MatMatMult so that I can
>>>> use standard eigenvalue solvers.
>>>>
>>>> Thanks,
>>>> Bikash
>>>> On May 11, 2015 1:00 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>>>>
>>>>> On Mon, May 11, 2015 at 8:47 AM, Bikash Kanungo <bikash at umich.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Patrick,
>>>>>>
>>>>>> Thanks for the clarification. I want it to be an efficient operation.
>>>>>> The idea is to convert a generalized eigenvalue problem (H*x = lamda*M*x)
>>>>>> to a standard one (M^{-1}*H*x = lambda*x). The M^{-1} is of type MATNEST
>>>>>> whereas H is MPIAIJ. In my problem M^{-1} is computed once whereas H
>>>>>> changes every iteration. So performing low-level matrix multiplication or
>>>>>> creating H as MATNEST and assembling it from its sub-matrices at every
>>>>>> iteration sounds inefficient.
>>>>>>
>>>>>
>>>>> You can solve these types of things with SLEPc without explicit MatMat.
>>>>>
>>>>>    Matt
>>>>>
>>>>>
>>>>>> Regards,
>>>>>> Bikash
>>>>>>
>>>>>> On Mon, May 11, 2015 at 9:21 AM, Patrick Sanan <
>>>>>> patrick.sanan at gmail.com> wrote:
>>>>>>
>>>>>>> MatMatMult with MATNEST does not seem to be supported, based only on
>>>>>>> the fact that there are no functions of the form MatMatMult_*_MatNest
>>>>>>> defined with the MATNEST implementation :
>>>>>>> http://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/nest/matnest.c.html
>>>>>>>
>>>>>>> Does this operation need to be efficient? (That is, are you forming
>>>>>>> this matrix for debugging or experimental purposes, or with the intention
>>>>>>> of using it within an efficient, scalable piece of code?). If not, it
>>>>>>> should be possible to use lower-level matrix operations as defined by the
>>>>>>> API to extract the appropriately-sized submatrices from A and (assuming
>>>>>>> that MatMatMult is defined between MATMPIAIJ and the submatrices of your
>>>>>>> Matnest), perform the matrix multiplications and additions "by hand" .
>>>>>>>
>>>>>>> On Mon, May 11, 2015 at 2:44 PM, Bikash Kanungo <bikash at umich.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have two matrices: A of type MPIAIJ and B of type MATNEST. Is
>>>>>>>> there any way to perform A*B after B has been assembled from its
>>>>>>>> sub-matrices?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bikash
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bikash S. Kanungo
>>>>>>>> PhD Student
>>>>>>>> Computational Materials Physics Group
>>>>>>>> Mechanical Engineering
>>>>>>>> University of Michigan
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bikash S. Kanungo
>>>>>> PhD Student
>>>>>> Computational Materials Physics Group
>>>>>> Mechanical Engineering
>>>>>> University of Michigan
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>
>
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/c398dbe5/attachment.html>

From jed at jedbrown.org  Mon May 11 12:58:36 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 May 2015 13:58:36 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<87r3qn57vp.fsf@jedbrown.org>
	<CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>
Message-ID: <87lhgv573n.fsf@jedbrown.org>

Bikash Kanungo <bikash at umich.edu> writes:

> The physics of the problem requires evaluation of M^{-1} from four
> sub-matrices. That's why I'm constrained to use it as MATNEST. The other
> option seems to be MatGetLocalSubMatrix but it seemed easier for me to go
> with MATNEST (particularly to avoid the local index sets).

If you need a matrix-matrix multiply, then artificial intermediate
scales will tend to reduce performance rather than improving it.

We can implement MatMatMult_Nest_Nest, but based on what you've said, I
doubt it's a good fit for your problem.  MatMatMult_Nest_MPIAIJ would
likely perform worse than MatMatMult_MPIAIJ_MPIAIJ.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/9e84d89e/attachment.pgp>

From bikash at umich.edu  Mon May 11 13:06:30 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 14:06:30 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <87lhgv573n.fsf@jedbrown.org>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<87r3qn57vp.fsf@jedbrown.org>
	<CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>
	<87lhgv573n.fsf@jedbrown.org>
Message-ID: <CA+H+vqLd9gFfxGNqUUVBHagY9NjY2xjfknsgTkq+e18GcAh+2A@mail.gmail.com>

So I guess, in terms of performance creating M^{-1} through
MatGetLocalSubMatrix and then explicitly build M^{-1}H is a more efficient
way.
On May 11, 2015 1:58 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Bikash Kanungo <bikash at umich.edu> writes:
>
> > The physics of the problem requires evaluation of M^{-1} from four
> > sub-matrices. That's why I'm constrained to use it as MATNEST. The other
> > option seems to be MatGetLocalSubMatrix but it seemed easier for me to go
> > with MATNEST (particularly to avoid the local index sets).
>
> If you need a matrix-matrix multiply, then artificial intermediate
> scales will tend to reduce performance rather than improving it.
>
> We can implement MatMatMult_Nest_Nest, but based on what you've said, I
> doubt it's a good fit for your problem.  MatMatMult_Nest_MPIAIJ would
> likely perform worse than MatMatMult_MPIAIJ_MPIAIJ.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/bfcf811a/attachment.html>

From jed at jedbrown.org  Mon May 11 13:09:59 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 May 2015 14:09:59 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <CA+H+vqLd9gFfxGNqUUVBHagY9NjY2xjfknsgTkq+e18GcAh+2A@mail.gmail.com>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<87r3qn57vp.fsf@jedbrown.org>
	<CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>
	<87lhgv573n.fsf@jedbrown.org>
	<CA+H+vqLd9gFfxGNqUUVBHagY9NjY2xjfknsgTkq+e18GcAh+2A@mail.gmail.com>
Message-ID: <87iobz56ko.fsf@jedbrown.org>

Bikash Kanungo <bikash at umich.edu> writes:

> So I guess, in terms of performance creating M^{-1} through
> MatGetLocalSubMatrix and then explicitly build M^{-1}H is a more efficient
> way.

Yes, unless this destroys sparsity that would otherwise be exploitable.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/29befb45/attachment.pgp>

From bikash at umich.edu  Mon May 11 13:11:24 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Mon, 11 May 2015 14:11:24 -0400
Subject: [petsc-users] MatMatMult involving MPIAIJ and MATNEST
In-Reply-To: <87iobz56ko.fsf@jedbrown.org>
References: <CA+H+vqLAMgEP+PDem+XkZx8spZHTBpwTnqK4gPfaMzVPUsfz0g@mail.gmail.com>
	<CA+z91Tf1tcM=-9ij7RcwYnMuuaZQ3W8YY5_+i8YNkKTXa=dJMw@mail.gmail.com>
	<CA+H+vqLLVuRAqmCfaJWw4woPHMoi65Gov2quFjFMnOMsm+uZQw@mail.gmail.com>
	<CAMYG4GkkS7KMazVZ-DcyHiuZ26=ZB6RPy+mqCV89vKWWH6hLog@mail.gmail.com>
	<CA+H+vqKzEAAPJ1r2yH62hGDGYDjtfg+b1aybqY5ZKdkF2SCmpw@mail.gmail.com>
	<CA+z91Te2-DQkZLAJtJt7hE8TmzgPqgb1ao0WniFo=9pu0hhcdQ@mail.gmail.com>
	<CA+H+vqKdLZ4t3QrRxpf9WNRzOZr40OzWMnWmm_sUbt7Q2GirJA@mail.gmail.com>
	<87r3qn57vp.fsf@jedbrown.org>
	<CA+H+vqJSer2u0mDJv6ojOTmsJ_=fLByoOPAuve4_1w4CKG3mDw@mail.gmail.com>
	<87lhgv573n.fsf@jedbrown.org>
	<CA+H+vqLd9gFfxGNqUUVBHagY9NjY2xjfknsgTkq+e18GcAh+2A@mail.gmail.com>
	<87iobz56ko.fsf@jedbrown.org>
Message-ID: <CA+H+vqJkVRUaL0-e9ky6ytKQ6a0gAmGp5CLmf+nF9B5H0gs0Mw@mail.gmail.com>

Cool. Thank you all for your valuable suggestions.
On May 11, 2015 2:10 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Bikash Kanungo <bikash at umich.edu> writes:
>
> > So I guess, in terms of performance creating M^{-1} through
> > MatGetLocalSubMatrix and then explicitly build M^{-1}H is a more
> efficient
> > way.
>
> Yes, unless this destroys sparsity that would otherwise be exploitable.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/750b83c5/attachment-0001.html>

From cctv19 at taiyangtech.com  Mon May 11 03:29:02 2015
From: cctv19 at taiyangtech.com (Tracy)
Date: Mon, 11 May 2015 16:29:02 +0800
Subject: [petsc-users] 4CH AHD DVR just 18.95usd/pc.
Message-ID: <138925906217CC9184F7D3976FD8714C9A9700D4EF@TAIYANGTECH.COM>

Dear friend:

How are you?

Glad to tell you that we have 4CH AHD DVR just 18.95usd/pc.

Contact us soon if you are  interested in it!

Email me if you don't want to received our email again!
 
BR
 
Tracy
 
2015-05-11



Sunivision Technology Development Co.,Ltd
Add:Floor 3, Building B, Taoyuan Industrial Park, Huangpu East Development Zone, Guangzhou, China 
Tel:+86 20 85915657+818  FAX:+86 20 28217656-803

Mobile/Viber/Whatsapp/Wechat:+86 15602392859
Alibaba Trade Manager:hscctv
QQ:1978294815
Skype:sunivision19
Website:www.sunivision.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/e31cae75/attachment.html>

From jychang48 at gmail.com  Mon May 11 15:53:37 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 11 May 2015 15:53:37 -0500
Subject: [petsc-users] MPI bindings for PETSc local and global numbering
Message-ID: <CAP2=TMi10sapCaBbHGaHQ7kuTsEdzRm7vW=Vu=cTN8UBR6c9kQ@mail.gmail.com>

Hi all,

I am experimenting with mpiexec.hydra's binding options for the following
compute node:

Machine (64GB total)
  NUMANode L#0 (P#0 32GB)
    Socket L#0 + L3 L#0 (25MB)
      L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0
(P#0)
      L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1
(P#1)
      L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2
(P#2)
      L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3
(P#3)
      L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4
(P#4)
      L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5
(P#5)
      L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6
(P#6)
      L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7
(P#7)
      L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8
(P#8)
      L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9
(P#9)
    HostBridge L#0
      PCIBridge
        PCI 1000:0087
          Block L#0 "sda"
      PCIBridge
        PCI 15b3:1003
          Net L#1 "eth0"
          Net L#2 "ib0"
          OpenFabrics L#3 "mlx4_0"
      PCIBridge
        PCI 8086:1521
          Net L#4 "eth1"
        PCI 8086:1521
          Net L#5 "eth2"
      PCIBridge
        PCI 102b:0533
      PCI 8086:1d02
  NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (25MB)
    L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU
L#10 (P#10)
    L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU
L#11 (P#11)
    L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU
L#12 (P#12)
    L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU
L#13 (P#13)
    L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU
L#14 (P#14)
    L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU
L#15 (P#15)
    L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU
L#16 (P#16)
    L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU
L#17 (P#17)
    L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU
L#18 (P#18)
    L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU
L#19 (P#19)

I am currently using up to 16 processes and have the following:

${MPIEXEC}.hydra -n <procs> -bind-to hwthread -map-by socket ./myprogram
<args>

For 16 processes, the binding looks like this:

process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 1 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0
process 2 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 3 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
process 4 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 5 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
process 6 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 7 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
process 8 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
process 10 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0
process 12 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
process 14 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

where the ranks are staggered (but evenly distributed) among the two
sockets. However, if I grouped the ranks sequentially like this:

process 0 binding: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 1 binding: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 2 binding: 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 3 binding: 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 4 binding: 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 5 binding: 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
process 6 binding: 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
process 7 binding: 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
process 8 binding: 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0
process 9 binding: 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
process 10 binding: 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
process 11 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
process 12 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0
process 13 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0
process 14 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
process 15 binding: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

Would this be more efficient? I was wondering if, in the context of
local/global numbering for PETSc's matrices and vectors, this would make
any difference. I have used ParMETIS to distribute/organize my DMPlex.

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/105926a6/attachment.html>

From jed at jedbrown.org  Mon May 11 16:07:51 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 May 2015 17:07:51 -0400
Subject: [petsc-users] MPI bindings for PETSc local and global numbering
In-Reply-To: <CAP2=TMi10sapCaBbHGaHQ7kuTsEdzRm7vW=Vu=cTN8UBR6c9kQ@mail.gmail.com>
References: <CAP2=TMi10sapCaBbHGaHQ7kuTsEdzRm7vW=Vu=cTN8UBR6c9kQ@mail.gmail.com>
Message-ID: <87bnhq6cwo.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:
> ${MPIEXEC}.hydra -n <procs> -bind-to hwthread -map-by socket ./myprogram
> <args>

Why are you using "-map-by socket" if you want a sequential ordering?

Abstractly, I expect a tradeoff between load balancing in case of
correlated irregularity and cache locality, but would tend to prefer
sequential ordering (so that nearby row numbers are more likely to share
cache).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/3aa48940/attachment.pgp>

From jychang48 at gmail.com  Mon May 11 16:22:38 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 11 May 2015 16:22:38 -0500
Subject: [petsc-users] MPI bindings for PETSc local and global numbering
In-Reply-To: <87bnhq6cwo.fsf@jedbrown.org>
References: <CAP2=TMi10sapCaBbHGaHQ7kuTsEdzRm7vW=Vu=cTN8UBR6c9kQ@mail.gmail.com>
	<87bnhq6cwo.fsf@jedbrown.org>
Message-ID: <CAP2=TMgU1Qw9am32xYpr_XUH8Q3xXOO8_yC__-tqg-ZSrbUuiQ@mail.gmail.com>

"-map-by socket" was for the non-sequential ordering I had. I got
sequentially ordering with "-bind-to
user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17"

On Mon, May 11, 2015 at 4:07 PM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
> > ${MPIEXEC}.hydra -n <procs> -bind-to hwthread -map-by socket ./myprogram
> > <args>
>
> Why are you using "-map-by socket" if you want a sequential ordering?
>
> Abstractly, I expect a tradeoff between load balancing in case of
> correlated irregularity and cache locality, but would tend to prefer
> sequential ordering (so that nearby row numbers are more likely to share
> cache).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/f4db320c/attachment.html>

From jed at jedbrown.org  Mon May 11 16:41:02 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 11 May 2015 17:41:02 -0400
Subject: [petsc-users] MPI bindings for PETSc local and global numbering
In-Reply-To: <CAP2=TMgU1Qw9am32xYpr_XUH8Q3xXOO8_yC__-tqg-ZSrbUuiQ@mail.gmail.com>
References: <CAP2=TMi10sapCaBbHGaHQ7kuTsEdzRm7vW=Vu=cTN8UBR6c9kQ@mail.gmail.com>
	<87bnhq6cwo.fsf@jedbrown.org>
	<CAP2=TMgU1Qw9am32xYpr_XUH8Q3xXOO8_yC__-tqg-ZSrbUuiQ@mail.gmail.com>
Message-ID: <878ucu6bdd.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> "-map-by socket" was for the non-sequential ordering I had. I got
> sequentially ordering with "-bind-to
> user:0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17"

  -map-by hwthread
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/79407308/attachment.pgp>

From mfadams at lbl.gov  Mon May 11 18:55:59 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 11 May 2015 16:55:59 -0700
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <876182a7z0.fsf@jedbrown.org>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
Message-ID: <CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>

More Titan fun.  I have built for PGI on Titan but tests are failing to
build: -lhwloc?

cc -mp -fast  -o ex56 ex56.o
-L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
-Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
-L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
-Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
-L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
-lhwloc -lssl -lcrypto -ldl -lstdc++
/usr/bin/ld: cannot find -lhwloc
make: [ex56] Error 2 (ignored)
/

On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
>
> > Also - Perhaps with intel compilers - the recommended blas is
> > something other than ACML? [like MKL?]. Something to check..
>
> MKL (and anything built using the Intel compiler) has the "run slower on
> AMD" feature.  For any given operation, ACML won't necessarily be
> faster, but at least it's not intentionally crippled.  For most of what
> PETSc does, any such MKL/ACML difference is irrelevant.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/87afdcb4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 4618599 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/87afdcb4/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 99132 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/87afdcb4/attachment-0003.obj>

From knepley at gmail.com  Mon May 11 19:04:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 11 May 2015 19:04:03 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
Message-ID: <CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>

On Mon, May 11, 2015 at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:

> More Titan fun.  I have built for PGI on Titan but tests are failing to
> build: -lhwloc?
>
> cc -mp -fast  -o ex56 ex56.o
> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
> -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
> -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
> -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
> -lhwloc -lssl -lcrypto -ldl -lstdc++
> /usr/bin/ld: cannot find -lhwloc
> make: [ex56] Error 2 (ignored)
> /
>

Barry had hwloc autoinstalling, but then we talked him out of it. You need
to clean things out and build again, since it
thinks that it is still there.

  Matt


> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Satish Balay <balay at mcs.anl.gov> writes:
>>
>> > Also - Perhaps with intel compilers - the recommended blas is
>> > something other than ACML? [like MKL?]. Something to check..
>>
>> MKL (and anything built using the Intel compiler) has the "run slower on
>> AMD" feature.  For any given operation, ACML won't necessarily be
>> faster, but at least it's not intentionally crippled.  For most of what
>> PETSc does, any such MKL/ACML difference is irrelevant.
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/5df82aab/attachment.html>

From mfadams at lbl.gov  Mon May 11 19:06:27 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 11 May 2015 17:06:27 -0700
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>
Message-ID: <CADOhEh55uNXY8SnRGJ5BAttCno52Z6v-3DjMOPSaOPO_WxBngw@mail.gmail.com>

I think I did a super clean build...  I will try again.

On Mon, May 11, 2015 at 5:04 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 11, 2015 at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> More Titan fun.  I have built for PGI on Titan but tests are failing to
>> build: -lhwloc?
>>
>> cc -mp -fast  -o ex56 ex56.o
>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
>> -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
>> -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
>> -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
>> -lhwloc -lssl -lcrypto -ldl -lstdc++
>> /usr/bin/ld: cannot find -lhwloc
>> make: [ex56] Error 2 (ignored)
>> /
>>
>
> Barry had hwloc autoinstalling, but then we talked him out of it. You need
> to clean things out and build again, since it
> thinks that it is still there.
>
>   Matt
>
>
>> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Satish Balay <balay at mcs.anl.gov> writes:
>>>
>>> > Also - Perhaps with intel compilers - the recommended blas is
>>> > something other than ACML? [like MKL?]. Something to check..
>>>
>>> MKL (and anything built using the Intel compiler) has the "run slower on
>>> AMD" feature.  For any given operation, ACML won't necessarily be
>>> faster, but at least it's not intentionally crippled.  For most of what
>>> PETSc does, any such MKL/ACML difference is irrelevant.
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/121f98cd/attachment.html>

From danyang.su at gmail.com  Mon May 11 19:10:53 2015
From: danyang.su at gmail.com (Danyang Su)
Date: Mon, 11 May 2015 17:10:53 -0700
Subject: [petsc-users] Is matrix analysis available in PETSc or external
	package?
Message-ID: <5551450D.4080308@gmail.com>

Hi All,

I recently have some time-dependent cases that have difficulty in 
convergence. It needs a lot of linear iterations during a specific time, 
e.g., more than 100 linear iterations for every newton iteration. In 
PETSc parallel version, this number will be doubled or even more. Our 
case usually needs less than 10 iterations for every newton iteration. 
This is NOT caused by PETSc relative routines as the original sequential 
solver (ILU) also has the same problem.

The results seems reasonable, but the large number of iterations leads 
me to suspect that something is not well implemented in a particular 
module. This module is developed a long time ago by another guy and it 
sounds impossible for me to check every line (thousands of lines) and do 
theoretical analysis.Does PETSc or external package provide matrix 
analysis to help to locate the 'suspected' entry?

Thanks and regards,

Danyang

From knepley at gmail.com  Mon May 11 19:14:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 11 May 2015 19:14:58 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh55uNXY8SnRGJ5BAttCno52Z6v-3DjMOPSaOPO_WxBngw@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>
	<CADOhEh55uNXY8SnRGJ5BAttCno52Z6v-3DjMOPSaOPO_WxBngw@mail.gmail.com>
Message-ID: <CAMYG4Gm=iGKT-CfwZYu5BXEn+2EYXSyT1AjQrYJA1_hfU3KKeA@mail.gmail.com>

On Mon, May 11, 2015 at 7:06 PM, Mark Adams <mfadams at lbl.gov> wrote:

> I think I did a super clean build...  I will try again.
>

I did not look at the configure.log. If this is master, it should not
download hwloc automatically. If its not, turn it off.

   Matt


> On Mon, May 11, 2015 at 5:04 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Mon, May 11, 2015 at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> More Titan fun.  I have built for PGI on Titan but tests are failing to
>>> build: -lhwloc?
>>>
>>> cc -mp -fast  -o ex56 ex56.o
>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
>>> -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
>>> -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
>>> -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
>>> -lhwloc -lssl -lcrypto -ldl -lstdc++
>>> /usr/bin/ld: cannot find -lhwloc
>>> make: [ex56] Error 2 (ignored)
>>> /
>>>
>>
>> Barry had hwloc autoinstalling, but then we talked him out of it. You
>> need to clean things out and build again, since it
>> thinks that it is still there.
>>
>>   Matt
>>
>>
>>> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Satish Balay <balay at mcs.anl.gov> writes:
>>>>
>>>> > Also - Perhaps with intel compilers - the recommended blas is
>>>> > something other than ACML? [like MKL?]. Something to check..
>>>>
>>>> MKL (and anything built using the Intel compiler) has the "run slower on
>>>> AMD" feature.  For any given operation, ACML won't necessarily be
>>>> faster, but at least it's not intentionally crippled.  For most of what
>>>> PETSc does, any such MKL/ACML difference is irrelevant.
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/f56fda30/attachment.html>

From mfadams at lbl.gov  Mon May 11 19:35:48 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 11 May 2015 17:35:48 -0700
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CAMYG4Gm=iGKT-CfwZYu5BXEn+2EYXSyT1AjQrYJA1_hfU3KKeA@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>
	<CADOhEh55uNXY8SnRGJ5BAttCno52Z6v-3DjMOPSaOPO_WxBngw@mail.gmail.com>
	<CAMYG4Gm=iGKT-CfwZYu5BXEn+2EYXSyT1AjQrYJA1_hfU3KKeA@mail.gmail.com>
Message-ID: <CADOhEh59NB11QLpHnDD0nY7kRYDq61hNySN=L-Vt+9c1F0bR1A@mail.gmail.com>

I did a fresh build again. No luck.  Is this how I turn it off?

  '--download-hwloc=0',



On Mon, May 11, 2015 at 5:14 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 11, 2015 at 7:06 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> I think I did a super clean build...  I will try again.
>>
>
> I did not look at the configure.log. If this is master, it should not
> download hwloc automatically. If its not, turn it off.
>
>    Matt
>
>
>> On Mon, May 11, 2015 at 5:04 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Mon, May 11, 2015 at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> More Titan fun.  I have built for PGI on Titan but tests are failing to
>>>> build: -lhwloc?
>>>>
>>>> cc -mp -fast  -o ex56 ex56.o
>>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
>>>> -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
>>>> -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
>>>> -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
>>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
>>>> -lhwloc -lssl -lcrypto -ldl -lstdc++
>>>> /usr/bin/ld: cannot find -lhwloc
>>>> make: [ex56] Error 2 (ignored)
>>>> /
>>>>
>>>
>>> Barry had hwloc autoinstalling, but then we talked him out of it. You
>>> need to clean things out and build again, since it
>>> thinks that it is still there.
>>>
>>>   Matt
>>>
>>>
>>>> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Satish Balay <balay at mcs.anl.gov> writes:
>>>>>
>>>>> > Also - Perhaps with intel compilers - the recommended blas is
>>>>> > something other than ACML? [like MKL?]. Something to check..
>>>>>
>>>>> MKL (and anything built using the Intel compiler) has the "run slower
>>>>> on
>>>>> AMD" feature.  For any given operation, ACML won't necessarily be
>>>>> faster, but at least it's not intentionally crippled.  For most of what
>>>>> PETSc does, any such MKL/ACML difference is irrelevant.
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/03c08187/attachment-0001.html>

From knepley at gmail.com  Mon May 11 19:39:12 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 11 May 2015 19:39:12 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh59NB11QLpHnDD0nY7kRYDq61hNySN=L-Vt+9c1F0bR1A@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<CAMYG4G=9Vh8j4T2aJTVbfjEM+fsnai_EsAt7bDbsRvpKoArVPA@mail.gmail.com>
	<CADOhEh55uNXY8SnRGJ5BAttCno52Z6v-3DjMOPSaOPO_WxBngw@mail.gmail.com>
	<CAMYG4Gm=iGKT-CfwZYu5BXEn+2EYXSyT1AjQrYJA1_hfU3KKeA@mail.gmail.com>
	<CADOhEh59NB11QLpHnDD0nY7kRYDq61hNySN=L-Vt+9c1F0bR1A@mail.gmail.com>
Message-ID: <CAMYG4GmpxVOtzv-RpVjHj=rtnqBgkaphacUyYbFUM8zOxpDOsg@mail.gmail.com>

On Mon, May 11, 2015 at 7:35 PM, Mark Adams <mfadams at lbl.gov> wrote:

> I did a fresh build again. No luck.  Is this how I turn it off?
>
>   '--download-hwloc=0',
>

--with-hwloc=0

   Matt


> On Mon, May 11, 2015 at 5:14 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Mon, May 11, 2015 at 7:06 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> I think I did a super clean build...  I will try again.
>>>
>>
>> I did not look at the configure.log. If this is master, it should not
>> download hwloc automatically. If its not, turn it off.
>>
>>    Matt
>>
>>
>>> On Mon, May 11, 2015 at 5:04 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, May 11, 2015 at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> More Titan fun.  I have built for PGI on Titan but tests are failing
>>>>> to build: -lhwloc?
>>>>>
>>>>> cc -mp -fast  -o ex56 ex56.o
>>>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc
>>>>> -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib
>>>>> -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas
>>>>> -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib
>>>>> -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis
>>>>> -lhwloc -lssl -lcrypto -ldl -lstdc++
>>>>> /usr/bin/ld: cannot find -lhwloc
>>>>> make: [ex56] Error 2 (ignored)
>>>>> /
>>>>>
>>>>
>>>> Barry had hwloc autoinstalling, but then we talked him out of it. You
>>>> need to clean things out and build again, since it
>>>> thinks that it is still there.
>>>>
>>>>   Matt
>>>>
>>>>
>>>>> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>>> Satish Balay <balay at mcs.anl.gov> writes:
>>>>>>
>>>>>> > Also - Perhaps with intel compilers - the recommended blas is
>>>>>> > something other than ACML? [like MKL?]. Something to check..
>>>>>>
>>>>>> MKL (and anything built using the Intel compiler) has the "run slower
>>>>>> on
>>>>>> AMD" feature.  For any given operation, ACML won't necessarily be
>>>>>> faster, but at least it's not intentionally crippled.  For most of
>>>>>> what
>>>>>> PETSc does, any such MKL/ACML difference is irrelevant.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/b608b24c/attachment.html>

From bsmith at mcs.anl.gov  Mon May 11 21:01:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 11 May 2015 21:01:34 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
Message-ID: <A793353C-C3EC-4556-B877-68691B5D6D89@mcs.anl.gov>


> On May 11, 2015, at 6:55 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> More Titan fun.  I have built for PGI on Titan but tests are failing to build: -lhwloc?

  Mark,

  What test is this? Did "make tests" run correctly?  How did you run this thing below, remember if you cut and paste everything including what you typed and the directory that your are in that helps. Did you happen to do this make test on a different login node or with a different configuration? 

> 
> cc -mp -fast  -o ex56 ex56.o -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib  -lpetsc -Wl,-rpath,/opt/pgi/14.2.0/linux86/5.1/lib -L/opt/pgi/14.2.0/linux86/5.1/lib -llapack -lblas -Wl,-rpath,/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -L/autofs/na3_home1/adams/petsc_master_opt64idx_pgi/lib -lparmetis -lmetis -lhwloc -lssl -lcrypto -ldl -lstdc++
> /usr/bin/ld: cannot find -lhwloc
> make: [ex56] Error 2 (ignored)

  Notes: ./configure ran fine and detected -lhwloc in some standard system install location under normal circumstances it couldn't just disappear for a different example.


> /
> 
> On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> Satish Balay <balay at mcs.anl.gov> writes:
> 
> > Also - Perhaps with intel compilers - the recommended blas is
> > something other than ACML? [like MKL?]. Something to check..
> 
> MKL (and anything built using the Intel compiler) has the "run slower on
> AMD" feature.  For any given operation, ACML won't necessarily be
> faster, but at least it's not intentionally crippled.  For most of what
> PETSc does, any such MKL/ACML difference is irrelevant.
> 
> <configure.log><make.log>


From hzhang at mcs.anl.gov  Mon May 11 21:19:58 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Mon, 11 May 2015 21:19:58 -0500
Subject: [petsc-users] Is matrix analysis available in PETSc or external
	package?
In-Reply-To: <5551450D.4080308@gmail.com>
References: <5551450D.4080308@gmail.com>
Message-ID: <CAGCphBvKA-ncsnGBNEE3N7QSGDkgcqVMJr-Z3AAG+3p4bJ6Ogg@mail.gmail.com>

Danyang:
>
>
> I recently have some time-dependent cases that have difficulty in
> convergence. It needs a lot of linear iterations during a specific time,
> e.g., more than 100 linear iterations for every newton iteration. In PETSc
> parallel version, this number will be doubled or even more. Our case
> usually needs less than 10 iterations for every newton iteration. This is
> NOT caused by PETSc relative routines as the original sequential solver
> (ILU) also has the same problem.
>
> The results seems reasonable, but the large number of iterations leads me
> to suspect that something is not well implemented in a particular module.
> This module is developed a long time ago by another guy and it sounds
> impossible for me to check every line (thousands of lines) and do
> theoretical analysis.Does PETSc or external package provide matrix analysis
> to help to locate the 'suspected' entry?
>

We need know what solvers being used. My guess is the default
gmres/bjacobi/ilu(0). Please run your code with option '-ts_view' or
'-snes_view' to find out.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150511/7fab54c9/attachment.html>

From mfadams at lbl.gov  Tue May 12 11:12:55 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 12 May 2015 09:12:55 -0700
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <A793353C-C3EC-4556-B877-68691B5D6D89@mcs.anl.gov>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<A793353C-C3EC-4556-B877-68691B5D6D89@mcs.anl.gov>
Message-ID: <CADOhEh6JJAYKn-nD2OptwRdM-W645SCEmOoBG_5nG_xr7HZf2Q@mail.gmail.com>

>
>   Notes: ./configure ran fine and detected -lhwloc in some standard system
> install location under normal circumstances it couldn't just disappear for
> a different example.
>
>
I configured in an interactive shell, so on a compute node.  I tried to
'make ex56' on a login node, as usual.  So I am guessing that it would have
worked if I had made it a compute node.  They have an inconstancy, I'm
guessing. I can try it all on a compute node ....

Mark


>
> > /
> >
> > On Fri, May 8, 2015 at 5:51 PM, Jed Brown <jed at jedbrown.org> wrote:
> > Satish Balay <balay at mcs.anl.gov> writes:
> >
> > > Also - Perhaps with intel compilers - the recommended blas is
> > > something other than ACML? [like MKL?]. Something to check..
> >
> > MKL (and anything built using the Intel compiler) has the "run slower on
> > AMD" feature.  For any given operation, ACML won't necessarily be
> > faster, but at least it's not intentionally crippled.  For most of what
> > PETSc does, any such MKL/ACML difference is irrelevant.
> >
> > <configure.log><make.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/98bf3020/attachment.html>

From balay at mcs.anl.gov  Tue May 12 11:15:29 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 12 May 2015 11:15:29 -0500
Subject: [petsc-users] configure error on Titan with Intel
In-Reply-To: <CADOhEh6JJAYKn-nD2OptwRdM-W645SCEmOoBG_5nG_xr7HZf2Q@mail.gmail.com>
References: <CADOhEh6ZJYgZpp_G1fL08XFpiWUUNfXOnqhwbvmYUesj+krX6w@mail.gmail.com>
	<alpine.LFD.2.11.1505081044220.2737@asterix>
	<CADOhEh7J+dGfKhEg-b8n_q0qu8zXrB0TJWfSjEgemnCCY20oUA@mail.gmail.com>
	<alpine.LFD.2.11.1505081108330.2737@asterix>
	<alpine.LFD.2.11.1505081115200.2737@asterix>
	<876182a7z0.fsf@jedbrown.org>
	<CADOhEh5H60DydOzo936a+KQZwt-uVMyQ17AHf2b18nJthAKQ=g@mail.gmail.com>
	<A793353C-C3EC-4556-B877-68691B5D6D89@mcs.anl.gov>
	<CADOhEh6JJAYKn-nD2OptwRdM-W645SCEmOoBG_5nG_xr7HZf2Q@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505121115001.14218@asterix>

On Tue, 12 May 2015, Mark Adams wrote:

> >
> >   Notes: ./configure ran fine and detected -lhwloc in some standard system
> > install location under normal circumstances it couldn't just disappear for
> > a different example.
> >
> >
> I configured in an interactive shell, so on a compute node.  I tried to
> 'make ex56' on a login node, as usual.  So I am guessing that it would have
> worked if I had made it a compute node.  They have an inconstancy, I'm
> guessing.

yes.

> I can try it all on a compute node ....

Use --with-hwloc=0 as Matt suggested..

Satish


From bsmith at mcs.anl.gov  Tue May 12 13:13:24 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 12 May 2015 13:13:24 -0500
Subject: [petsc-users] Is matrix analysis available in PETSc or external
	package?
In-Reply-To: <5551450D.4080308@gmail.com>
References: <5551450D.4080308@gmail.com>
Message-ID: <413298C5-1B25-47FF-93E3-EB455C436643@mcs.anl.gov>


> On May 11, 2015, at 7:10 PM, Danyang Su <danyang.su at gmail.com> wrote:
> 
> Hi All,
> 
> I recently have some time-dependent cases that have difficulty in convergence. It needs a lot of linear iterations during a specific time, e.g., more than 100 linear iterations for every newton iteration. In PETSc parallel version, this number will be doubled or even more. Our case usually needs less than 10 iterations for every newton iteration. This is NOT caused by PETSc relative routines as the original sequential solver (ILU) also has the same problem.
> 
> The results seems reasonable, but the large number of iterations leads me to suspect that something is not well implemented in a particular module. This module is developed a long time ago by another guy and it sounds impossible for me to check every line (thousands of lines) and do theoretical analysis.Does PETSc or external package provide matrix analysis to help to locate the 'suspected' entry?

   What do you mean by 'matrix analaysis'?   You can run with -ksp_view_mat binary to have all the matrices saved in a binary file called binaryoutput which you can read into MATLAB (with $PETSC_DIR/shared/petsc/matlab/PetscBinaryRead.m) or python (with bin/PetscBinaryIO.py). You can then look at the matrices, determine the largest entry whatever you wish.

   But it is also very possible that the due to changes in the model you are evolving in time the linear system simple does get more ill-conditioned during part of the simulation. This is pretty common and need not indicate that there is anything wrong with the model or the code. 

  Barry

> 
> Thanks and regards,
> 
> Danyang


From danyang.su at gmail.com  Tue May 12 13:31:15 2015
From: danyang.su at gmail.com (Danyang Su)
Date: Tue, 12 May 2015 11:31:15 -0700
Subject: [petsc-users] Is matrix analysis available in PETSc or external
 package?
In-Reply-To: <413298C5-1B25-47FF-93E3-EB455C436643@mcs.anl.gov>
References: <5551450D.4080308@gmail.com>
	<413298C5-1B25-47FF-93E3-EB455C436643@mcs.anl.gov>
Message-ID: <555246F3.8050602@gmail.com>

On 15-05-12 11:13 AM, Barry Smith wrote:
>> On May 11, 2015, at 7:10 PM, Danyang Su <danyang.su at gmail.com> wrote:
>>
>> Hi All,
>>
>> I recently have some time-dependent cases that have difficulty in convergence. It needs a lot of linear iterations during a specific time, e.g., more than 100 linear iterations for every newton iteration. In PETSc parallel version, this number will be doubled or even more. Our case usually needs less than 10 iterations for every newton iteration. This is NOT caused by PETSc relative routines as the original sequential solver (ILU) also has the same problem.
>>
>> The results seems reasonable, but the large number of iterations leads me to suspect that something is not well implemented in a particular module. This module is developed a long time ago by another guy and it sounds impossible for me to check every line (thousands of lines) and do theoretical analysis.Does PETSc or external package provide matrix analysis to help to locate the 'suspected' entry?
>     What do you mean by 'matrix analaysis'?   You can run with -ksp_view_mat binary to have all the matrices saved in a binary file called binaryoutput which you can read into MATLAB (with $PETSC_DIR/shared/petsc/matlab/PetscBinaryRead.m) or python (with bin/PetscBinaryIO.py). You can then look at the matrices, determine the largest entry whatever you wish.
>
>     But it is also very possible that the due to changes in the model you are evolving in time the linear system simple does get more ill-conditioned during part of the simulation. This is pretty common and need not indicate that there is anything wrong with the model or the code.
>
>    Barry
Hi Barry,

Here 'matrix analysis' means some extreme large entries in the matrix 
that causes the linear solver difficult to converge. Since I am not 
pretty sure if anything is wrong with the code or not, I just want to 
check the location of these entries and how it is calculated. This is a 
complex reactive transport problem so it is possible that nothing is 
wrong but the system is more ill conditioned in as specific simulation 
time.

I also have the matrix output to matrix market format that I can use to 
analysis.

Thanks,

Danyang
>
>> Thanks and regards,
>>
>> Danyang


From danyang.su at gmail.com  Tue May 12 13:35:07 2015
From: danyang.su at gmail.com (Danyang Su)
Date: Tue, 12 May 2015 11:35:07 -0700
Subject: [petsc-users] Is matrix analysis available in PETSc or external
 package?
In-Reply-To: <CAGCphBvKA-ncsnGBNEE3N7QSGDkgcqVMJr-Z3AAG+3p4bJ6Ogg@mail.gmail.com>
References: <5551450D.4080308@gmail.com>
	<CAGCphBvKA-ncsnGBNEE3N7QSGDkgcqVMJr-Z3AAG+3p4bJ6Ogg@mail.gmail.com>
Message-ID: <555247DB.2010906@gmail.com>

On 15-05-11 07:19 PM, Hong wrote:
> Danyang:
>
>
>     I recently have some time-dependent cases that have difficulty in
>     convergence. It needs a lot of linear iterations during a specific
>     time, e.g., more than 100 linear iterations for every newton
>     iteration. In PETSc parallel version, this number will be doubled
>     or even more. Our case usually needs less than 10 iterations for
>     every newton iteration. This is NOT caused by PETSc relative
>     routines as the original sequential solver (ILU) also has the same
>     problem.
>
>     The results seems reasonable, but the large number of iterations
>     leads me to suspect that something is not well implemented in a
>     particular module. This module is developed a long time ago by
>     another guy and it sounds impossible for me to check every line
>     (thousands of lines) and do theoretical analysis.Does PETSc or
>     external package provide matrix analysis to help to locate the
>     'suspected' entry?
>
> We need know what solvers being used. My guess is the default
> gmres/bjacobi/ilu(0). Please run your code with option '-ts_view' or 
> '-snes_view' to find out.
Hi Hong,

Sorry for the late reply. I guess there is nothing wrong in the solver 
or data structure, but maybe some large entries in the linear system 
that cause the problem hard to solve. I will export the matrix to check 
these values.

Thanks,

Danyang
>
> Hong
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/958f5071/attachment.html>

From mlohry at princeton.edu  Tue May 12 15:38:33 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Tue, 12 May 2015 16:38:33 -0400
Subject: [petsc-users] segfault on MatCreateSNESMF ?
In-Reply-To: <87zj5tse8k.fsf@jedbrown.org>
References: <553A85DF.5000400@gmail.com>
	<4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov>
	<553AA10B.5090208@gmail.com> <553AA4B1.5010603@princeton.edu>
	<D8AF2E4C-FC2A-4CF8-91AC-D034416C87A5@mcs.anl.gov>
	<553AB0C6.5010503@princeton.edu>
	<B534BBB4-D1A5-4080-935D-1944B41C9DF3@mcs.anl.gov>
	<553E5AD5.1060208@princeton.edu> <87383ltvf8.fsf@jedbrown.org>
	<553E5FFF.6080905@princeton.edu> <87zj5tse8k.fsf@jedbrown.org>
Message-ID: <555264C9.5010303@princeton.edu>

I'm getting a segfault when trying to set up a matrix-free solver:

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaabe7b7de in MatCreateSNESMF (snes=0x0, J=0x7fffffffc340)
     at /home/mlohry/dev/petsc-3.5.3/src/snes/mf/snesmfj.c:151
151      if (snes->vec_func) {
(gdb) bt
#0  0x00002aaaabe7b7de in MatCreateSNESMF (snes=0x0, J=0x7fffffffc340)
     at /home/mlohry/dev/petsc-3.5.3/src/snes/mf/snesmfj.c:151


from the line


   SNES snes; // non-linear algebraic solver context
   TSGetSNES(ts,&snes);
   Mat JacMatFree=NULL;
   MatCreateSNESMF(snes,&JacMatFree); // tell non-linear solver this is 
matrix-free


any idea where this could come from?

From mlohry at princeton.edu  Tue May 12 15:45:08 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Tue, 12 May 2015 16:45:08 -0400
Subject: [petsc-users] segfault on MatCreateSNESMF ?
In-Reply-To: <555264C9.5010303@princeton.edu>
References: <553A85DF.5000400@gmail.com>	<4DD6A0C4-34B3-4E4D-997D-0BE285081380@mcs.anl.gov>	<553AA10B.5090208@gmail.com>
	<553AA4B1.5010603@princeton.edu>	<D8AF2E4C-FC2A-4CF8-91AC-D034416C87A5@mcs.anl.gov>	<553AB0C6.5010503@princeton.edu>	<B534BBB4-D1A5-4080-935D-1944B41C9DF3@mcs.anl.gov>	<553E5AD5.1060208@princeton.edu>
	<87383ltvf8.fsf@jedbrown.org>	<553E5FFF.6080905@princeton.edu>
	<87zj5tse8k.fsf@jedbrown.org> <555264C9.5010303@princeton.edu>
Message-ID: <55526654.4050306@princeton.edu>

sorry, disregard that. I had forgotten to do TSCreate earlier.

On 05/12/2015 04:38 PM, Mark Lohry wrote:
> I'm getting a segfault when trying to set up a matrix-free solver:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00002aaaabe7b7de in MatCreateSNESMF (snes=0x0, J=0x7fffffffc340)
>     at /home/mlohry/dev/petsc-3.5.3/src/snes/mf/snesmfj.c:151
> 151      if (snes->vec_func) {
> (gdb) bt
> #0  0x00002aaaabe7b7de in MatCreateSNESMF (snes=0x0, J=0x7fffffffc340)
>     at /home/mlohry/dev/petsc-3.5.3/src/snes/mf/snesmfj.c:151
>
>
> from the line
>
>
>   SNES snes; // non-linear algebraic solver context
>   TSGetSNES(ts,&snes);
>   Mat JacMatFree=NULL;
>   MatCreateSNESMF(snes,&JacMatFree); // tell non-linear solver this is 
> matrix-free
>
>
> any idea where this could come from?


From mrosso at uci.edu  Tue May 12 19:55:51 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Tue, 12 May 2015 17:55:51 -0700
Subject: [petsc-users] Hypre + openMP
Message-ID: <1431478551.3820.1.camel@enterprise-A>

 Hi,

is it possible to use the openmp capabilities of hypre via petsc?
Thanks,

Michele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/5ab4b76e/attachment.html>

From bsmith at mcs.anl.gov  Tue May 12 21:35:13 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 12 May 2015 21:35:13 -0500
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <1431478551.3820.1.camel@enterprise-A>
References: <1431478551.3820.1.camel@enterprise-A>
Message-ID: <4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>


  You could compile hypre yourself with the OpenMP feature turned on and then configure PETSc to use that version of hypre; of course the rest of the PETSc code would not utilize the OpenMP threads. 

   Barry

BTW: I don't know of any evidence that using hypre with OpenMP is superior to using hypre without so I aside from academic curiosity I don't think there would be a reason to do this.

> On May 12, 2015, at 7:55 PM, Michele Rosso <mrosso at uci.edu> wrote:
> 
> Hi,
> 
> is it possible to use the openmp capabilities of hypre via petsc?
> Thanks,
> 
> Michele


From mrosso at uci.edu  Tue May 12 23:41:13 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Tue, 12 May 2015 21:41:13 -0700
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
Message-ID: <CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>

Barry,

thanks for your answer. The reason I'm asking is that multigrid limits the
number of MPI tasks I can use for a given grid, since k multigrid levels
require at least 2^(k-1) grid nodes per direction. I was wondering if using
OpenMP together with MPI could help circumventing the problem. If you have
any other suggestions, it would be greatly appreciated.

Best,
Michele

  You could compile hypre yourself with the OpenMP feature turned on and
then configure PETSc to use that version of hypre; of course the rest of
the PETSc code would not utilize the OpenMP threads.

   Barry

BTW: I don't know of any evidence that using hypre with OpenMP is superior
to using hypre without so I aside from academic curiosity I don't think
there would be a reason to do this.

> On May 12, 2015, at 7:55 PM, Michele Rosso <mrosso at uci.edu> wrote:
>
> Hi,
>
> is it possible to use the openmp capabilities of hypre via petsc?
> Thanks,
>
> Michele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/1299f1e7/attachment-0001.html>

From knepley at gmail.com  Tue May 12 23:50:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 12 May 2015 23:50:19 -0500
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
Message-ID: <CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>

On Tue, May 12, 2015 at 11:41 PM, Michele Rosso <mrosso at uci.edu> wrote:

> Barry,
>
> thanks for your answer. The reason I'm asking is that multigrid limits the
> number of MPI tasks I can use for a given grid, since k multigrid levels
> require at least 2^(k-1) grid nodes per direction. I was wondering if using
> OpenMP together with MPI could help circumventing the problem. If you have
> any other suggestions, it would be greatly appreciated.
>
This is only for naive partitioning. Good AMG necks down the processor set
correctly, for example GAMG in PETSc.

  Thanks,

     Matt


> Best,
> Michele
>
>   You could compile hypre yourself with the OpenMP feature turned on and
> then configure PETSc to use that version of hypre; of course the rest of
> the PETSc code would not utilize the OpenMP threads.
>
>    Barry
>
> BTW: I don't know of any evidence that using hypre with OpenMP is superior
> to using hypre without so I aside from academic curiosity I don't think
> there would be a reason to do this.
>
> > On May 12, 2015, at 7:55 PM, Michele Rosso <mrosso at uci.edu> wrote:
> >
> > Hi,
> >
> > is it possible to use the openmp capabilities of hypre via petsc?
> > Thanks,
> >
> > Michele
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/5af77756/attachment.html>

From mrosso at uci.edu  Wed May 13 00:01:49 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Tue, 12 May 2015 22:01:49 -0700
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
	<CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
Message-ID: <CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>

Thanks Matt,
Could you give me an example of not-native partitioning? I have a cubic
domain and a 3D domain decomposition. I use dmda3d to create the
partitioning.

Thanks,
Michele
On May 12, 2015 9:51 PM, "Matthew Knepley" <knepley at gmail.com> wrote:

> On Tue, May 12, 2015 at 11:41 PM, Michele Rosso <mrosso at uci.edu> wrote:
>
>> Barry,
>>
>> thanks for your answer. The reason I'm asking is that multigrid limits
>> the number of MPI tasks I can use for a given grid, since k multigrid
>> levels require at least 2^(k-1) grid nodes per direction. I was wondering
>> if using OpenMP together with MPI could help circumventing the problem. If
>> you have any other suggestions, it would be greatly appreciated.
>>
> This is only for naive partitioning. Good AMG necks down the processor set
> correctly, for example GAMG in PETSc.
>
>   Thanks,
>
>      Matt
>
>
>> Best,
>> Michele
>>
>>   You could compile hypre yourself with the OpenMP feature turned on and
>> then configure PETSc to use that version of hypre; of course the rest of
>> the PETSc code would not utilize the OpenMP threads.
>>
>>    Barry
>>
>> BTW: I don't know of any evidence that using hypre with OpenMP is
>> superior to using hypre without so I aside from academic curiosity I don't
>> think there would be a reason to do this.
>>
>> > On May 12, 2015, at 7:55 PM, Michele Rosso <mrosso at uci.edu> wrote:
>> >
>> > Hi,
>> >
>> > is it possible to use the openmp capabilities of hypre via petsc?
>> > Thanks,
>> >
>> > Michele
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/9a1e3406/attachment.html>

From knepley at gmail.com  Wed May 13 00:05:33 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 13 May 2015 00:05:33 -0500
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
	<CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
	<CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
Message-ID: <CAMYG4Gk6gA6hpnKF=QvTFBo94GfPkayQNOFXgWXuHWmHSgUj-A@mail.gmail.com>

On Wed, May 13, 2015 at 12:01 AM, Michele Rosso <mrosso at uci.edu> wrote:

> Thanks Matt,
> Could you give me an example of not-native partitioning? I have a cubic
> domain and a 3D domain decomposition. I use dmda3d to create the
> partitioning.
>
"naive", and I was talking about coarse grids. Coarse grids should be
partitioned onto fewer than P processes.

   Matt


> Thanks,
>
> Michele
> On May 12, 2015 9:51 PM, "Matthew Knepley" <knepley at gmail.com> wrote:
>
>> On Tue, May 12, 2015 at 11:41 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>
>>> Barry,
>>>
>>> thanks for your answer. The reason I'm asking is that multigrid limits
>>> the number of MPI tasks I can use for a given grid, since k multigrid
>>> levels require at least 2^(k-1) grid nodes per direction. I was wondering
>>> if using OpenMP together with MPI could help circumventing the problem. If
>>> you have any other suggestions, it would be greatly appreciated.
>>>
>> This is only for naive partitioning. Good AMG necks down the processor
>> set correctly, for example GAMG in PETSc.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Best,
>>> Michele
>>>
>>>   You could compile hypre yourself with the OpenMP feature turned on and
>>> then configure PETSc to use that version of hypre; of course the rest of
>>> the PETSc code would not utilize the OpenMP threads.
>>>
>>>    Barry
>>>
>>> BTW: I don't know of any evidence that using hypre with OpenMP is
>>> superior to using hypre without so I aside from academic curiosity I don't
>>> think there would be a reason to do this.
>>>
>>> > On May 12, 2015, at 7:55 PM, Michele Rosso <mrosso at uci.edu> wrote:
>>> >
>>> > Hi,
>>> >
>>> > is it possible to use the openmp capabilities of hypre via petsc?
>>> > Thanks,
>>> >
>>> > Michele
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/8e8a2d3e/attachment.html>

From jed at jedbrown.org  Wed May 13 00:19:10 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 12 May 2015 23:19:10 -0600
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
	<CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
	<CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
Message-ID: <87h9rhawc1.fsf@jedbrown.org>

Michele Rosso <mrosso at uci.edu> writes:

> Thanks Matt,
> Could you give me an example of not-native partitioning? I have a cubic
> domain and a 3D domain decomposition. I use dmda3d to create the
> partitioning.

Are you talking about geometric multigrid (AMG coursening is typically
irregular)?  Anyway, this is just math: what does it mean to have less
than one degree of freedom on a level?  Geometric MG on DMDA does have
requirements, but if you're using Hypre, I don't see why you'd care
about that.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/3080b6f8/attachment.pgp>

From mrosso at uci.edu  Wed May 13 00:29:22 2015
From: mrosso at uci.edu (Michele Rosso)
Date: Tue, 12 May 2015 22:29:22 -0700
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <87h9rhawc1.fsf@jedbrown.org>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
	<CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
	<CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
	<87h9rhawc1.fsf@jedbrown.org>
Message-ID: <CAN1seh_vQCbmVXS1XgV_vE1UrMp-Fx+0WG6gUtqWs3GGgAZ9ug@mail.gmail.com>

Hi Jed,

I see what you mean. I am using geometric multigrid and therefore I have a
limit on the number of processors I can use. I was looking for an
alternative and found hypre, but I did not realize that since it's
algebraic multigrid there is no such limitation.
Anyway, let's say I want to keep the geometric multigrid: is there a way to
work around the problem?

Thanks,
Michele
 On May 12, 2015 10:20 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Michele Rosso <mrosso at uci.edu> writes:
>
> > Thanks Matt,
> > Could you give me an example of not-native partitioning? I have a cubic
> > domain and a 3D domain decomposition. I use dmda3d to create the
> > partitioning.
>
> Are you talking about geometric multigrid (AMG coursening is typically
> irregular)?  Anyway, this is just math: what does it mean to have less
> than one degree of freedom on a level?  Geometric MG on DMDA does have
> requirements, but if you're using Hypre, I don't see why you'd care
> about that.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150512/42c65290/attachment.html>

From dave.mayhem23 at gmail.com  Wed May 13 01:39:02 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Wed, 13 May 2015 08:39:02 +0200
Subject: [petsc-users] Hypre + openMP
In-Reply-To: <CAN1seh_vQCbmVXS1XgV_vE1UrMp-Fx+0WG6gUtqWs3GGgAZ9ug@mail.gmail.com>
References: <1431478551.3820.1.camel@enterprise-A>
	<4130744D-B656-4682-B937-A26BCFE3E17D@mcs.anl.gov>
	<CAN1seh8P4d5c9uAf1Sbvz8ZY23sKP9Z1wQGVzsdtCCnHh-RWjA@mail.gmail.com>
	<CAMYG4G=B4Gh5HS5qOC85TVOiRRoYfsetTOhQx3MOpmRoiGLoHg@mail.gmail.com>
	<CAN1seh9KyN77MfjrJ9=Uvm_hRpsPfYYp16CujuFHZy2NG5wiQw@mail.gmail.com>
	<87h9rhawc1.fsf@jedbrown.org>
	<CAN1seh_vQCbmVXS1XgV_vE1UrMp-Fx+0WG6gUtqWs3GGgAZ9ug@mail.gmail.com>
Message-ID: <CAJ98EDoq5TkUpjne_dWTQ5NKw-0qh06Z2mwfgHKOy5refSjzmg@mail.gmail.com>

Yes, coarsen the dmda until you hit your processor limitation, then use
preonly + hypre/GAMG as your smoother on the coarsest geometric multi grid
level.

Cheers,
  Dave

On Wednesday, 13 May 2015, Michele Rosso <mrosso at uci.edu> wrote:

> Hi Jed,
>
> I see what you mean. I am using geometric multigrid and therefore I have a
> limit on the number of processors I can use. I was looking for an
> alternative and found hypre, but I did not realize that since it's
> algebraic multigrid there is no such limitation.
> Anyway, let's say I want to keep the geometric multigrid: is there a way
> to work around the problem?
>
> Thanks,
> Michele
>  On May 12, 2015 10:20 PM, "Jed Brown" <jed at jedbrown.org
> <javascript:_e(%7B%7D,'cvml','jed at jedbrown.org');>> wrote:
>
>> Michele Rosso <mrosso at uci.edu
>> <javascript:_e(%7B%7D,'cvml','mrosso at uci.edu');>> writes:
>>
>> > Thanks Matt,
>> > Could you give me an example of not-native partitioning? I have a cubic
>> > domain and a 3D domain decomposition. I use dmda3d to create the
>> > partitioning.
>>
>> Are you talking about geometric multigrid (AMG coursening is typically
>> irregular)?  Anyway, this is just math: what does it mean to have less
>> than one degree of freedom on a level?  Geometric MG on DMDA does have
>> requirements, but if you're using Hypre, I don't see why you'd care
>> about that.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/35a2ae9a/attachment-0001.html>

From venkateshgk.j at gmail.com  Wed May 13 10:28:42 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Wed, 13 May 2015 20:58:42 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
Message-ID: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>

Dear All,

I am having a block matrix Eigenvalue problem AX = lambda BX

My B is singular, it is a block matrix and only one block has non-zero
values.

Matlab Eigs is giving the result.

I am using option: "./ex7 -f1 A1 -f2 B1 -st_type sinvert"

Result is: Detected zero pivot in LU factorization.

The ex7 is an example from SLEPc.

Pls advice what to do ?

cheers,
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/0092d430/attachment.html>

From jroman at dsic.upv.es  Wed May 13 10:32:17 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 13 May 2015 17:32:17 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
Message-ID: <5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>


El 13/05/2015, a las 17:28, venkatesh g escribi?:

> Dear All,
> 
> I am having a block matrix Eigenvalue problem AX = lambda BX
> 
> My B is singular, it is a block matrix and only one block has non-zero values. 
> 
> Matlab Eigs is giving the result. 
> 
> I am using option: "./ex7 -f1 A1 -f2 B1 -st_type sinvert"  
> 
> Result is: Detected zero pivot in LU factorization.
> 
> The ex7 is an example from SLEPc. 
> 
> Pls advice what to do ? 
> 
> cheers,
> Venkatesh
> 
> 

If you know where your wanted eigenvalues are, set the target value, for instance:
./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5

Jose


From jychang48 at gmail.com  Wed May 13 14:54:01 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 13 May 2015 14:54:01 -0500
Subject: [petsc-users] Using matrix-free method for SNES problems with an
	explicit Jacobian
Message-ID: <CAP2=TMjog7hf_ghHtBmMQ_Ugvk7wfDGQVAtBvrJD9QZSrh+0vg@mail.gmail.com>

Hello everyone,

>From what I read online, one could use a matrix-free method if using the
conjugate gradient method for a symmetric and positive definite problem.
Some of the PETSc SNES examples like ex12 compute the Jacobian A
explicitly, but I was wondering if it's possible to convert this
problem into matrix free form. That is, to directly assemble the product of
A*x into vector form without first forming A and then invoking MatMult().

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/6658111f/attachment.html>

From knepley at gmail.com  Wed May 13 15:15:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 13 May 2015 15:15:09 -0500
Subject: [petsc-users] Using matrix-free method for SNES problems with
 an explicit Jacobian
In-Reply-To: <CAP2=TMjog7hf_ghHtBmMQ_Ugvk7wfDGQVAtBvrJD9QZSrh+0vg@mail.gmail.com>
References: <CAP2=TMjog7hf_ghHtBmMQ_Ugvk7wfDGQVAtBvrJD9QZSrh+0vg@mail.gmail.com>
Message-ID: <CAMYG4GnoT3YoyV7n74d5dwcGTQ1P6aTXp6+omAZjbsqHLZXbJA@mail.gmail.com>

On Wed, May 13, 2015 at 2:54 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Hello everyone,
>
> From what I read online, one could use a matrix-free method if using the
> conjugate gradient method for a symmetric and positive definite problem.
> Some of the PETSc SNES examples like ex12 compute the Jacobian A
> explicitly, but I was wondering if it's possible to convert this
> problem into matrix free form. That is, to directly assemble the product of
> A*x into vector form without first forming A and then invoking MatMult().
>

Yes, Jed advocates this all the time, but there is some infrastructure that
you want for this
to be really effective. For quadratic and higher orders, it makes sense to
compute the action
of your operator MF. This is done by defining something that looks a lot
like your residual
evaluation (element vec in and out) but computes the linearization.
However, you still usually
need a low order explicit matrix for preconditioning. I have not spent time
hooking that all together.

  Thanks,

    Matt


> Thanks,
> Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/fa3db454/attachment.html>

From gbisht at lbl.gov  Wed May 13 16:49:19 2015
From: gbisht at lbl.gov (Gautam Bisht)
Date: Wed, 13 May 2015 14:49:19 -0700
Subject: [petsc-users] Question regrading SNES tutorial ex28
Message-ID: <CAPz1Tneed5gY3q87VU30vAeGhTiLYUQuZ=0G0A1UsXNcnpvU0w@mail.gmail.com>

Hi PETSc team,

Any insights into why the residual equation for U & K are scaled by dx in
this example (line no. 48-49 and 68 of in ex28.c
<https://bitbucket.org/petsc/petsc/src/8db46702c31aaef455dabea2fa9d7dd897381b5d/src/snes/examples/tutorials/ex28.c?at=master#cl-8>
)?

Additionally, the comment regarding algebraic equation is incorrect.

 * Algebraic (K):

 *     exp(k-1) + k = u + 1/(1/(1+u) + 1/(1+u_x^2))"


should be changed to

 * Algebraic (K):

 *     exp(k-1) + k = 1/(1/(1+u) + 1/(1+u_x^2))"


Thanks,
-Gautam.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/92084399/attachment.html>

From jychang48 at gmail.com  Wed May 13 16:59:45 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Wed, 13 May 2015 16:59:45 -0500
Subject: [petsc-users] Using matrix-free method for SNES problems with
 an explicit Jacobian
In-Reply-To: <CAMYG4GnoT3YoyV7n74d5dwcGTQ1P6aTXp6+omAZjbsqHLZXbJA@mail.gmail.com>
References: <CAP2=TMjog7hf_ghHtBmMQ_Ugvk7wfDGQVAtBvrJD9QZSrh+0vg@mail.gmail.com>
	<CAMYG4GnoT3YoyV7n74d5dwcGTQ1P6aTXp6+omAZjbsqHLZXbJA@mail.gmail.com>
Message-ID: <CAP2=TMjX8jWj8uJcg9kLPLo6VRJiRGvbjmvY9Z48si_54qRG1Q@mail.gmail.com>

Okay that makes sense. Reason for asking this is because I am solving the
following convex optimization problem for a linear diffusion problem:

min 1/2 x^T * A * x + x^T * g
s.t. 0 <= x <= 1

Using Tao's BLMVM solver, the objective function and gradient vector reads
as follows:

f = 0.5*x^T*A*x + x^T*(r - A*x_0)
g = A*(x - x_0) + r

Where A, x_0, and r are the jacobian, initial guess, and residual
respectively. To compute these, I have TaoSetObjectiveAndGradientRoutine()
invoke the following function:

#undef __FUNCT__
#define __FUNCT__ "FormFunctionGradient"
PetscErrorCode FormFunctionGradient(Tao tao, Vec x, PetscReal *f, Vec g,
void *ctx)
{
  AppCtx *user = (AppCtx*)ctx;
  PetscScalar       xtHx;
  PetscErrorCode    ierr;

  PetscFunctionBegin;
  ierr = MatMult(user->A,x,g);CHKERRQ(ierr);
  ierr = VecDot(x,g,&xtHx);CHKERRQ(ierr);
  ierr = VecDot(x,user->f,f);CHKERRQ(ierr);
  *f += 0.5*xtHx;
  ierr = VecAXPY(g,1.0,user->f);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

where user->f = (r - A*x_0)  The timing results from -log_summary seem to
indicate that MatMult(user->A,x,g) is a bottleneck within TaoSolve() so I
was wondering if I could somehow make the computation of Vec g matrix-free
considering that user->A is constant. The problem proliferates if I decide
to use TRON. Or is this unavoidable (Jason any thoughts?)

Thanks,
Justin

On Wed, May 13, 2015 at 3:15 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 13, 2015 at 2:54 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hello everyone,
>>
>> From what I read online, one could use a matrix-free method if using the
>> conjugate gradient method for a symmetric and positive definite problem.
>> Some of the PETSc SNES examples like ex12 compute the Jacobian A
>> explicitly, but I was wondering if it's possible to convert this
>> problem into matrix free form. That is, to directly assemble the product of
>> A*x into vector form without first forming A and then invoking MatMult().
>>
>
> Yes, Jed advocates this all the time, but there is some infrastructure
> that you want for this
> to be really effective. For quadratic and higher orders, it makes sense to
> compute the action
> of your operator MF. This is done by defining something that looks a lot
> like your residual
> evaluation (element vec in and out) but computes the linearization.
> However, you still usually
> need a low order explicit matrix for preconditioning. I have not spent
> time hooking that all together.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Justin
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/521ef37d/attachment.html>

From knepley at gmail.com  Wed May 13 17:05:40 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 13 May 2015 17:05:40 -0500
Subject: [petsc-users] Using matrix-free method for SNES problems with
 an explicit Jacobian
In-Reply-To: <CAP2=TMjX8jWj8uJcg9kLPLo6VRJiRGvbjmvY9Z48si_54qRG1Q@mail.gmail.com>
References: <CAP2=TMjog7hf_ghHtBmMQ_Ugvk7wfDGQVAtBvrJD9QZSrh+0vg@mail.gmail.com>
	<CAMYG4GnoT3YoyV7n74d5dwcGTQ1P6aTXp6+omAZjbsqHLZXbJA@mail.gmail.com>
	<CAP2=TMjX8jWj8uJcg9kLPLo6VRJiRGvbjmvY9Z48si_54qRG1Q@mail.gmail.com>
Message-ID: <CAMYG4G=2w_EcKcWj_VRZYjQGHwk+cxqSfii9EV8X=fsBas-0Tg@mail.gmail.com>

On Wed, May 13, 2015 at 4:59 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Okay that makes sense. Reason for asking this is because I am solving the
> following convex optimization problem for a linear diffusion problem:
>
> min 1/2 x^T * A * x + x^T * g
> s.t. 0 <= x <= 1
>
> Using Tao's BLMVM solver, the objective function and gradient vector reads
> as follows:
>
> f = 0.5*x^T*A*x + x^T*(r - A*x_0)
> g = A*(x - x_0) + r
>
> Where A, x_0, and r are the jacobian, initial guess, and residual
> respectively. To compute these, I have TaoSetObjectiveAndGradientRoutine()
> invoke the following function:
>
> #undef __FUNCT__
> #define __FUNCT__ "FormFunctionGradient"
> PetscErrorCode FormFunctionGradient(Tao tao, Vec x, PetscReal *f, Vec g,
> void *ctx)
> {
>   AppCtx *user = (AppCtx*)ctx;
>   PetscScalar       xtHx;
>   PetscErrorCode    ierr;
>
>   PetscFunctionBegin;
>   ierr = MatMult(user->A,x,g);CHKERRQ(ierr);
>   ierr = VecDot(x,g,&xtHx);CHKERRQ(ierr);
>   ierr = VecDot(x,user->f,f);CHKERRQ(ierr);
>   *f += 0.5*xtHx;
>   ierr = VecAXPY(g,1.0,user->f);CHKERRQ(ierr);
>   PetscFunctionReturn(0);
> }
>
> where user->f = (r - A*x_0)  The timing results from -log_summary seem to
> indicate that MatMult(user->A,x,g) is a bottleneck within TaoSolve() so I
> was wondering if I could somehow make the computation of Vec g matrix-free
> considering that user->A is constant. The problem proliferates if I decide
> to use TRON. Or is this unavoidable (Jason any thoughts?)
>

If you have a first order discretization, its still better to form the
matrix since it uses fewer flops on repeated application.

  Thanks,

     Matt


> Thanks,
> Justin
>
> On Wed, May 13, 2015 at 3:15 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Wed, May 13, 2015 at 2:54 PM, Justin Chang <jychang48 at gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> From what I read online, one could use a matrix-free method if using the
>>> conjugate gradient method for a symmetric and positive definite problem.
>>> Some of the PETSc SNES examples like ex12 compute the Jacobian A
>>> explicitly, but I was wondering if it's possible to convert this
>>> problem into matrix free form. That is, to directly assemble the product of
>>> A*x into vector form without first forming A and then invoking MatMult().
>>>
>>
>> Yes, Jed advocates this all the time, but there is some infrastructure
>> that you want for this
>> to be really effective. For quadratic and higher orders, it makes sense
>> to compute the action
>> of your operator MF. This is done by defining something that looks a lot
>> like your residual
>> evaluation (element vec in and out) but computes the linearization.
>> However, you still usually
>> need a low order explicit matrix for preconditioning. I have not spent
>> time hooking that all together.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>> Justin
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/305965f1/attachment-0001.html>

From jed at jedbrown.org  Wed May 13 17:18:05 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 13 May 2015 16:18:05 -0600
Subject: [petsc-users] Question regrading SNES tutorial ex28
In-Reply-To: <CAPz1Tneed5gY3q87VU30vAeGhTiLYUQuZ=0G0A1UsXNcnpvU0w@mail.gmail.com>
References: <CAPz1Tneed5gY3q87VU30vAeGhTiLYUQuZ=0G0A1UsXNcnpvU0w@mail.gmail.com>
Message-ID: <87a8x886le.fsf@jedbrown.org>

Gautam Bisht <gbisht at lbl.gov> writes:

> Hi PETSc team,
>
> Any insights into why the residual equation for U & K are scaled by dx in
> this example (line no. 48-49 and 68 of in ex28.c
> <https://bitbucket.org/petsc/petsc/src/8db46702c31aaef455dabea2fa9d7dd897381b5d/src/snes/examples/tutorials/ex28.c?at=master#cl-8>
> )?

The scaling by hx represents a volume integral, similar to what you
naturally get with finite element methods.  It helps keep the problem
well-scaled during grid refinement.

> Additionally, the comment regarding algebraic equation is incorrect.
>
>  * Algebraic (K):
>
>  *     exp(k-1) + k = u + 1/(1/(1+u) + 1/(1+u_x^2))"
>
>
> should be changed to
>
>  * Algebraic (K):
>
>  *     exp(k-1) + k = 1/(1/(1+u) + 1/(1+u_x^2))"

Thanks; I'll push this shortly.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150513/192a2f10/attachment.pgp>

From Manishkumar.K at LntTechservices.com  Thu May 14 01:27:15 2015
From: Manishkumar.K at LntTechservices.com (Manish Kumar K)
Date: Thu, 14 May 2015 06:27:15 +0000
Subject: [petsc-users] Reg : PETSc Installation
Message-ID: <AA7F30F772C09F47AE19F45AD9B68B7855159548@POCITMSEXMB06.LntUniverse.com>

Dear PETSc team,

                Greetings of the day.
I am facing error while installing PETSc on Windows using Cygwin shell/terminal .

Following is the steps followed for the installation from Cygwin shell

Method 1:

1)cd c:/cygwin64/package/petsc-3.3-p7      //PETSc top level directory
2) ./configure --with-cl=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich            //command to configure and install PETSc


I am getting an error on this commands invoked in Cygwin

Error message : C compiler you provided with -with-cc=gcc does not work

Method 2:

./configure --with-cc=cl --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich



Error message : C compiler you provided with -with-cc=cl does not work

Method 3:

./configure --with-cc='win32fe cl'  --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-fblaslapack



Error message : C compiler you provided with -with-cc=win32fe cl does not work


Any help on this would be of great value to me and my customers.

Best Regards
Manish K

L&T Technology Services Ltd

www.LntTechservices.com<http://www.lnttechservices.com/>

This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/ba9ad0a3/attachment.html>

From venkateshgk.j at gmail.com  Thu May 14 01:52:04 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Thu, 14 May 2015 12:22:04 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
Message-ID: <CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>

Hi Jose, all,

I tried that too, it is telling the same error of "Detected Zero pivot in
LU factorization".
My matrix B in AX=lambda BX has large no. of zeros and is singular.

Is there another way ?

Venkatesh

On Wed, May 13, 2015 at 9:02 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 13/05/2015, a las 17:28, venkatesh g escribi?:
>
> > Dear All,
> >
> > I am having a block matrix Eigenvalue problem AX = lambda BX
> >
> > My B is singular, it is a block matrix and only one block has non-zero
> values.
> >
> > Matlab Eigs is giving the result.
> >
> > I am using option: "./ex7 -f1 A1 -f2 B1 -st_type sinvert"
> >
> > Result is: Detected zero pivot in LU factorization.
> >
> > The ex7 is an example from SLEPc.
> >
> > Pls advice what to do ?
> >
> > cheers,
> > Venkatesh
> >
> >
>
> If you know where your wanted eigenvalues are, set the target value, for
> instance:
> ./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/2647ce52/attachment.html>

From Manishkumar.K at LntTechservices.com  Thu May 14 02:03:12 2015
From: Manishkumar.K at LntTechservices.com (Manish Kumar K)
Date: Thu, 14 May 2015 07:03:12 +0000
Subject: [petsc-users] FW: Reg : PETSc Installation
In-Reply-To: <AA7F30F772C09F47AE19F45AD9B68B785515958F@POCITMSEXMB06.LntUniverse.com>
References: <AA7F30F772C09F47AE19F45AD9B68B785515958F@POCITMSEXMB06.LntUniverse.com>
Message-ID: <AA7F30F772C09F47AE19F45AD9B68B78551595A2@POCITMSEXMB06.LntUniverse.com>

Dear PETSc team,

                Greetings of the day.
I am facing error while installing PETSc on Windows using Cygwin shell/terminal .

Following is the steps followed for the installation from Cygwin shell

Method 1:

1)cd c:/cygwin64/package/petsc-3.3-p7      //PETSc top level directory
2) ./configure --with-cl=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich            //command to configure and install PETSc


I am getting an error on this commands invoked in Cygwin

Error message : C compiler you provided with -with-cc=gcc does not work

Method 2:

./configure --with-cc=cl --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich



Error message : C compiler you provided with -with-cc=cl does not work

Method 3:

./configure --with-cc='win32fe cl'  --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-fblaslapack



Error message : C compiler you provided with -with-cc=win32fe cl does not work

I have attached a log file on this


Any help on this would be of great value to me and my customers.

Best Regards
Manish K

L&T Technology Services Ltd

www.LntTechservices.com<http://www.lnttechservices.com/>

This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/3e9866ba/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 59982 bytes
Desc: configure.log
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/3e9866ba/attachment-0001.obj>

From jroman at dsic.upv.es  Thu May 14 03:33:16 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 10:33:16 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
Message-ID: <8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>

Are you calling EPSSetFromOptions in your source code? 
Is your A-matrix also singular with a nullspace in common with B?
You can try with MUMPS as explained in section 3.4.1 of the users manual.
Also check that your matrices are the same as in Matlab.
Jose


El 14/05/2015, a las 08:52, venkatesh g escribi?:

> Hi Jose, all,
> 
> I tried that too, it is telling the same error of "Detected Zero pivot in LU factorization". 
> My matrix B in AX=lambda BX has large no. of zeros and is singular. 
> 
> Is there another way ? 
> 
> Venkatesh
> 
> On Wed, May 13, 2015 at 9:02 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> El 13/05/2015, a las 17:28, venkatesh g escribi?:
> 
> > Dear All,
> >
> > I am having a block matrix Eigenvalue problem AX = lambda BX
> >
> > My B is singular, it is a block matrix and only one block has non-zero values.
> >
> > Matlab Eigs is giving the result.
> >
> > I am using option: "./ex7 -f1 A1 -f2 B1 -st_type sinvert"
> >
> > Result is: Detected zero pivot in LU factorization.
> >
> > The ex7 is an example from SLEPc.
> >
> > Pls advice what to do ?
> >
> > cheers,
> > Venkatesh
> >
> >
> 
> If you know where your wanted eigenvalues are, set the target value, for instance:
> ./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5
> 
> Jose
> 
> 


From bsmith at mcs.anl.gov  Thu May 14 06:55:15 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 14 May 2015 06:55:15 -0500
Subject: [petsc-users] [petsc-maint] FW: Reg : PETSc Installation
In-Reply-To: <AA7F30F772C09F47AE19F45AD9B68B78551595A2@POCITMSEXMB06.LntUniverse.com>
References: <AA7F30F772C09F47AE19F45AD9B68B785515958F@POCITMSEXMB06.LntUniverse.com>
	<AA7F30F772C09F47AE19F45AD9B68B78551595A2@POCITMSEXMB06.LntUniverse.com>
Message-ID: <5C1B33BF-AB09-41AC-8938-E8D60A1BF0AD@mcs.anl.gov>


  From the log file it attempted to compile a trivial C program and the compiler returned a nonzero error code

/package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.o -I/cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers   /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.c 
Executing: /package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.o -I/cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers   /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.c 
sh: 
Possible ERROR while running compiler: ret = 50176
Source:
#include "confdefs.h"
#include "conffix.h"

int main() {
;
  return 0;
}

   So try to compile a trivial code in the same way manually and see what happens. Make a file conftest.c with 
int main() {
;
  return 0;
}

and then run 

/package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o conftest.o conftest.c 
echo $?

cut and paste everything you get back and send it.




> On May 14, 2015, at 2:03 AM, Manish Kumar K <Manishkumar.K at LntTechservices.com> wrote:
> 
> Dear PETSc team,
>   
>                 Greetings of the day.
> I am facing error while installing PETSc on Windows using Cygwin shell/terminal .
>  
> Following is the steps followed for the installation from Cygwin shell
>  
> Method 1: 
>  
> 1)cd c:/cygwin64/package/petsc-3.3-p7      //PETSc top level directory
> 2) ./configure --with-cl=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich            //command to configure and install PETSc       
>  
>  
> I am getting an error on this commands invoked in Cygwin
>                                                                       
> Error message : C compiler you provided with -with-cc=gcc does not work
>  
> Method 2: 
>  
> ./configure --with-cc=cl --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich
>  
>  
>                                                                       
> Error message : C compiler you provided with -with-cc=cl does not work
>  
> Method 3: 
>  
> ./configure --with-cc='win32fe cl'  --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-fblaslapack                                                                                                   
>  
>  
>                                                                       
> Error message : C compiler you provided with -with-cc=win32fe cl does not work
>  
> I have attached a log file on this
>  
>  
> Any help on this would be of great value to me and my customers.
>  
> Best Regards
> Manish K
> L&T Technology Services Ltd
> 
> www.LntTechservices.com
> 
> This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
> 
> <configure.log>


From vijay.gopal.c at gmail.com  Thu May 14 09:08:00 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Thu, 14 May 2015 16:08:00 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
Message-ID: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>

Hi,

I am using SLEPc to get the lowest m eigenvalues of a large sparse
hermitian matrix using Krylov-Schur.
The problem is that while asking for a large number of eigenvalues (e.g.
300) for a large problem
consisting of a 91454220 X 91454220 system, the solver does not give the
lowest 300 eigenvalues.
Instead it gives the following arbitrary negative values followed by a few
correct ones.

   -4733.882429           0.999888
>    -4729.554413             1.0006
>    -4725.841715           0.999887
>    -4702.457861            1.00061
>    -4700.608105           0.999887
>    -4691.142733           0.999885
>       -6.361168            1.18973
>       -6.361102        6.05898e-09
>       -6.361099           6.71e-09
>       -6.361093        7.66471e-09
>       -6.361082            1.17659



Although the solver says that it has successfully converged the 300
eigenvalues !
Please find the relevant files attached.

Could anyone tell me what I'm doing wrong ?

thanks,
 Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/ed9a9dd1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: problem.c
Type: text/x-csrc
Size: 11716 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/ed9a9dd1/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output
Type: application/octet-stream
Size: 12857 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/ed9a9dd1/attachment-0001.obj>

From jroman at dsic.upv.es  Thu May 14 10:17:31 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 17:17:31 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
Message-ID: <BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>


On 14/05/2015, Vijay Gopal Chilkuri wrote:

> Hi,
> 
> I am using SLEPc to get the lowest m eigenvalues of a large sparse hermitian matrix using Krylov-Schur.
> The problem is that while asking for a large number of eigenvalues (e.g. 300) for a large problem
> consisting of a 91454220 X 91454220 system, the solver does not give the lowest 300 eigenvalues.
> Instead it gives the following arbitrary negative values followed by a few correct ones.
> 
>    -4733.882429           0.999888
>    -4729.554413             1.0006
>    -4725.841715           0.999887
>    -4702.457861            1.00061
>    -4700.608105           0.999887
>    -4691.142733           0.999885
>       -6.361168            1.18973
>       -6.361102        6.05898e-09
>       -6.361099           6.71e-09
>       -6.361093        7.66471e-09
>       -6.361082            1.17659
> 
> 
> Although the solver says that it has successfully converged the 300 eigenvalues !
> Please find the relevant files attached.
> 
> Could anyone tell me what I'm doing wrong ?
> 
> thanks,
>  Vijay
> 
> <problem.c><output>

I do not see anything strange, apart from passing a pointer to long int in an argument of type PetscInt*. To guarantee portability, it would be better if all variables in your program are of type PetscInt, PetscMPIIInt, PetscReal, PetscScalar, etc.

I do not know what is going on. Do you get the same behaviour for smaller matrix sizes? Do you get the same problem when computing only a few eigenvalues?

Jose



From vijay.gopal.c at gmail.com  Thu May 14 10:28:52 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Thu, 14 May 2015 17:28:52 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
Message-ID: <CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>

I get the proper eigenvalues (and quite fast) when I ask for only 10-15 of
the lowest eigenvalues.
The only problem is when I ask for m ~ 100-300 of the lowest eigenvalues.

Is it possible that asking for large number of eigenvalues would cause
problems ?

I've got the same problem with smaller matrices of size 540540. Again for
this system, if I ask for
10-15 eigenvalues I get the proper correct converged eigenvalues, but when
I ask for more than 100 eigevalues
I get incorrect results !

If you would like to test I can send you the proper files.

Thanks,
 Vijay




On Thu, May 14, 2015 at 5:17 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> On 14/05/2015, Vijay Gopal Chilkuri wrote:
>
> > Hi,
> >
> > I am using SLEPc to get the lowest m eigenvalues of a large sparse
> hermitian matrix using Krylov-Schur.
> > The problem is that while asking for a large number of eigenvalues (e.g.
> 300) for a large problem
> > consisting of a 91454220 X 91454220 system, the solver does not give the
> lowest 300 eigenvalues.
> > Instead it gives the following arbitrary negative values followed by a
> few correct ones.
> >
> >    -4733.882429           0.999888
> >    -4729.554413             1.0006
> >    -4725.841715           0.999887
> >    -4702.457861            1.00061
> >    -4700.608105           0.999887
> >    -4691.142733           0.999885
> >       -6.361168            1.18973
> >       -6.361102        6.05898e-09
> >       -6.361099           6.71e-09
> >       -6.361093        7.66471e-09
> >       -6.361082            1.17659
> >
> >
> > Although the solver says that it has successfully converged the 300
> eigenvalues !
> > Please find the relevant files attached.
> >
> > Could anyone tell me what I'm doing wrong ?
> >
> > thanks,
> >  Vijay
> >
> > <problem.c><output>
>
> I do not see anything strange, apart from passing a pointer to long int in
> an argument of type PetscInt*. To guarantee portability, it would be better
> if all variables in your program are of type PetscInt, PetscMPIIInt,
> PetscReal, PetscScalar, etc.
>
> I do not know what is going on. Do you get the same behaviour for smaller
> matrix sizes? Do you get the same problem when computing only a few
> eigenvalues?
>
> Jose
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/60765595/attachment.html>

From jroman at dsic.upv.es  Thu May 14 10:34:00 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 17:34:00 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
Message-ID: <DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>


El 14/05/2015, a las 17:28, Vijay Gopal Chilkuri escribi?:

> I get the proper eigenvalues (and quite fast) when I ask for only 10-15 of the lowest eigenvalues. 
> The only problem is when I ask for m ~ 100-300 of the lowest eigenvalues.
> 
> Is it possible that asking for large number of eigenvalues would cause problems ?

It should not be a problem, but you could try running with -eps_mpd 50 to see if it helps.

> 
> I've got the same problem with smaller matrices of size 540540. Again for this system, if I ask for
> 10-15 eigenvalues I get the proper correct converged eigenvalues, but when I ask for more than 100 eigevalues
> I get incorrect results !
> 
> If you would like to test I can send you the proper files.

Yes, send them to me.

> 
> Thanks,
>  Vijay
> 
> 
> 
> 
> On Thu, May 14, 2015 at 5:17 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> On 14/05/2015, Vijay Gopal Chilkuri wrote:
> 
> > Hi,
> >
> > I am using SLEPc to get the lowest m eigenvalues of a large sparse hermitian matrix using Krylov-Schur.
> > The problem is that while asking for a large number of eigenvalues (e.g. 300) for a large problem
> > consisting of a 91454220 X 91454220 system, the solver does not give the lowest 300 eigenvalues.
> > Instead it gives the following arbitrary negative values followed by a few correct ones.
> >
> >    -4733.882429           0.999888
> >    -4729.554413             1.0006
> >    -4725.841715           0.999887
> >    -4702.457861            1.00061
> >    -4700.608105           0.999887
> >    -4691.142733           0.999885
> >       -6.361168            1.18973
> >       -6.361102        6.05898e-09
> >       -6.361099           6.71e-09
> >       -6.361093        7.66471e-09
> >       -6.361082            1.17659
> >
> >
> > Although the solver says that it has successfully converged the 300 eigenvalues !
> > Please find the relevant files attached.
> >
> > Could anyone tell me what I'm doing wrong ?
> >
> > thanks,
> >  Vijay
> >
> > <problem.c><output>
> 
> I do not see anything strange, apart from passing a pointer to long int in an argument of type PetscInt*. To guarantee portability, it would be better if all variables in your program are of type PetscInt, PetscMPIIInt, PetscReal, PetscScalar, etc.
> 
> I do not know what is going on. Do you get the same behaviour for smaller matrix sizes? Do you get the same problem when computing only a few eigenvalues?
> 
> Jose
> 
> 
> 


From balay at mcs.anl.gov  Thu May 14 10:35:27 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 14 May 2015 10:35:27 -0500
Subject: [petsc-users] [petsc-maint] FW: Reg : PETSc Installation
In-Reply-To: <5C1B33BF-AB09-41AC-8938-E8D60A1BF0AD@mcs.anl.gov>
References: <AA7F30F772C09F47AE19F45AD9B68B785515958F@POCITMSEXMB06.LntUniverse.com>
	<AA7F30F772C09F47AE19F45AD9B68B78551595A2@POCITMSEXMB06.LntUniverse.com>
	<5C1B33BF-AB09-41AC-8938-E8D60A1BF0AD@mcs.anl.gov>
Message-ID: <alpine.LFD.2.11.1505141032560.2861@asterix>

Which compilers do you want to use? gnu or MS? [or will either work for you?]

Also - any perticular reason for using petsc-3.3 instead of petsc-3.5?

Also - to make sure cygwin is not causing these problems - I'll recommend:

1. reboot
2. do not start any cygwin processes
3. run cygwin setup.exe - and click 'next' - all the way to completion. [it will
run some scripts to rebase the installed packages at the end]

And then try rebuilding with gnu compilers  - and resend the log for this build.

Satish

On Thu, 14 May 2015, Barry Smith wrote:

> 
>   From the log file it attempted to compile a trivial C program and the compiler returned a nonzero error code
> 
> /package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.o -I/cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers   /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.c 
> Executing: /package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.o -I/cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers   /cygdrive/c/Users/20055335/AppData/Local/Temp/petsc-pTZ0rV/config.setCompilers/conftest.c 
> sh: 
> Possible ERROR while running compiler: ret = 50176
> Source:
> #include "confdefs.h"
> #include "conffix.h"
> 
> int main() {
> ;
>   return 0;
> }
> 
>    So try to compile a trivial code in the same way manually and see what happens. Make a file conftest.c with 
> int main() {
> ;
>   return 0;
> }
> 
> and then run 
> 
> /package/petsc-3.3-p7/bin/win32fe/win32fe cl -c -o conftest.o conftest.c 
> echo $?
> 
> cut and paste everything you get back and send it.
> 
> 
> 
> 
> > On May 14, 2015, at 2:03 AM, Manish Kumar K <Manishkumar.K at LntTechservices.com> wrote:
> > 
> > Dear PETSc team,
> >   
> >                 Greetings of the day.
> > I am facing error while installing PETSc on Windows using Cygwin shell/terminal .
> >  
> > Following is the steps followed for the installation from Cygwin shell
> >  
> > Method 1: 
> >  
> > 1)cd c:/cygwin64/package/petsc-3.3-p7      //PETSc top level directory
> > 2) ./configure --with-cl=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich            //command to configure and install PETSc       
> >  
> >  
> > I am getting an error on this commands invoked in Cygwin
> >                                                                       
> > Error message : C compiler you provided with -with-cc=gcc does not work
> >  
> > Method 2: 
> >  
> > ./configure --with-cc=cl --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich
> >  
> >  
> >                                                                       
> > Error message : C compiler you provided with -with-cc=cl does not work
> >  
> > Method 3: 
> >  
> > ./configure --with-cc='win32fe cl'  --with-fc='win32fe ifort' --with-cxx='win32fe cl' --download-fblaslapack                                                                                                   
> >  
> >  
> >                                                                       
> > Error message : C compiler you provided with -with-cc=win32fe cl does not work
> >  
> > I have attached a log file on this
> >  
> >  
> > Any help on this would be of great value to me and my customers.
> >  
> > Best Regards
> > Manish K
> > L&T Technology Services Ltd
> > 
> > www.LntTechservices.com
> > 
> > This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
> > 
> > <configure.log>
> 
> 


From vijay.gopal.c at gmail.com  Thu May 14 11:00:06 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Thu, 14 May 2015 18:00:06 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
Message-ID: <CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>

Ok I've sent a job with the -eps_mpd 50 option and I still get the same
absurd eigenvalues !

You already have the driver .c file (called problem.c I also attach this
file) I send you the makefile.

The subroutine unit_l1_ is a fortran subroutine which gives the number of
nonzero elements in 22 rows at once. This subroutine can be compiled in the
following way.

Firstly, it uses the IRPF90 library.
You can download the library from this github link:
https://github.com/scemama/irpf90.git
the binary needed "irpf90" should be in the bin directory, just update the
relevant PATH environment variables.

Second, get the codes that generate the unit_l1_ subroutine from the
following repository: https://github.com/v1j4y/slepc_version.git
*please change to branch slepc_testing !*

> make
> make irpf90.a


should give you the irpf90.a library nessessary to compile the problem.c
file, I attach the makefile to compile the problem.c file with this mail.
To compile the final executable, please do "*make problem*"

and that should be enough !
sorry that it is not simpler.

Please write if you run into problems.

Thanks,
 Vijay


On Thu, May 14, 2015 at 5:34 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 14/05/2015, a las 17:28, Vijay Gopal Chilkuri escribi?:
>
> > I get the proper eigenvalues (and quite fast) when I ask for only 10-15
> of the lowest eigenvalues.
> > The only problem is when I ask for m ~ 100-300 of the lowest eigenvalues.
> >
> > Is it possible that asking for large number of eigenvalues would cause
> problems ?
>
> It should not be a problem, but you could try running with -eps_mpd 50 to
> see if it helps.
>
> >
> > I've got the same problem with smaller matrices of size 540540. Again
> for this system, if I ask for
> > 10-15 eigenvalues I get the proper correct converged eigenvalues, but
> when I ask for more than 100 eigevalues
> > I get incorrect results !
> >
> > If you would like to test I can send you the proper files.
>
> Yes, send them to me.
>
> >
> > Thanks,
> >  Vijay
> >
> >
> >
> >
> > On Thu, May 14, 2015 at 5:17 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >
> > On 14/05/2015, Vijay Gopal Chilkuri wrote:
> >
> > > Hi,
> > >
> > > I am using SLEPc to get the lowest m eigenvalues of a large sparse
> hermitian matrix using Krylov-Schur.
> > > The problem is that while asking for a large number of eigenvalues
> (e.g. 300) for a large problem
> > > consisting of a 91454220 X 91454220 system, the solver does not give
> the lowest 300 eigenvalues.
> > > Instead it gives the following arbitrary negative values followed by a
> few correct ones.
> > >
> > >    -4733.882429           0.999888
> > >    -4729.554413             1.0006
> > >    -4725.841715           0.999887
> > >    -4702.457861            1.00061
> > >    -4700.608105           0.999887
> > >    -4691.142733           0.999885
> > >       -6.361168            1.18973
> > >       -6.361102        6.05898e-09
> > >       -6.361099           6.71e-09
> > >       -6.361093        7.66471e-09
> > >       -6.361082            1.17659
> > >
> > >
> > > Although the solver says that it has successfully converged the 300
> eigenvalues !
> > > Please find the relevant files attached.
> > >
> > > Could anyone tell me what I'm doing wrong ?
> > >
> > > thanks,
> > >  Vijay
> > >
> > > <problem.c><output>
> >
> > I do not see anything strange, apart from passing a pointer to long int
> in an argument of type PetscInt*. To guarantee portability, it would be
> better if all variables in your program are of type PetscInt, PetscMPIIInt,
> PetscReal, PetscScalar, etc.
> >
> > I do not know what is going on. Do you get the same behaviour for
> smaller matrix sizes? Do you get the same problem when computing only a few
> eigenvalues?
> >
> > Jose
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/b8f07426/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 191 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/b8f07426/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: problem.c
Type: text/x-csrc
Size: 11716 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/b8f07426/attachment-0001.c>

From jroman at dsic.upv.es  Thu May 14 12:04:19 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 19:04:19 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
Message-ID: <AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>

Ok. I was able to build the code, but I need a small problem in order to be able to run without having to access a big supercomputer.
Jose


El 14/05/2015, a las 18:00, Vijay Gopal Chilkuri escribi?:

> Ok I've sent a job with the -eps_mpd 50 option and I still get the same absurd eigenvalues !
> 
> You already have the driver .c file (called problem.c I also attach this file) I send you the makefile.
> 
> The subroutine unit_l1_ is a fortran subroutine which gives the number of nonzero elements in 22 rows at once. This subroutine can be compiled in the following way.
> 
> Firstly, it uses the IRPF90 library.
> You can download the library from this github link: https://github.com/scemama/irpf90.git
> the binary needed "irpf90" should be in the bin directory, just update the relevant PATH environment variables.
> 
> Second, get the codes that generate the unit_l1_ subroutine from the following repository: https://github.com/v1j4y/slepc_version.git
> please change to branch slepc_testing !
> make
> make irpf90.a
>  
> should give you the irpf90.a library nessessary to compile the problem.c file, I attach the makefile to compile the problem.c file with this mail.
> To compile the final executable, please do "make problem"
> 
> and that should be enough !
> sorry that it is not simpler.
> 
> Please write if you run into problems.
> 
> Thanks,
>  Vijay
> 


From venkateshgk.j at gmail.com  Thu May 14 12:13:48 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Thu, 14 May 2015 22:43:48 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
Message-ID: <CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>

Hi,
1. I am not calling EPSSetFromOptions in source code. I am using the ex7.c
given in the SLEPc webpage.
2. A-matrix is not singular. Only B is singular.
3. I will try with MUMPS.
4. Matrix sizes are same as in Matlab. If B is not singular it is giving
same result as Matlab, but if B is singular it gives the zero pivot issue
which Matlab doesnt give.
So is there a way ?

On Thu, May 14, 2015 at 2:03 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> Are you calling EPSSetFromOptions in your source code?
> Is your A-matrix also singular with a nullspace in common with B?
> You can try with MUMPS as explained in section 3.4.1 of the users manual.
> Also check that your matrices are the same as in Matlab.
> Jose
>
>
> El 14/05/2015, a las 08:52, venkatesh g escribi?:
>
> > Hi Jose, all,
> >
> > I tried that too, it is telling the same error of "Detected Zero pivot
> in LU factorization".
> > My matrix B in AX=lambda BX has large no. of zeros and is singular.
> >
> > Is there another way ?
> >
> > Venkatesh
> >
> > On Wed, May 13, 2015 at 9:02 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >
> > El 13/05/2015, a las 17:28, venkatesh g escribi?:
> >
> > > Dear All,
> > >
> > > I am having a block matrix Eigenvalue problem AX = lambda BX
> > >
> > > My B is singular, it is a block matrix and only one block has non-zero
> values.
> > >
> > > Matlab Eigs is giving the result.
> > >
> > > I am using option: "./ex7 -f1 A1 -f2 B1 -st_type sinvert"
> > >
> > > Result is: Detected zero pivot in LU factorization.
> > >
> > > The ex7 is an example from SLEPc.
> > >
> > > Pls advice what to do ?
> > >
> > > cheers,
> > > Venkatesh
> > >
> > >
> >
> > If you know where your wanted eigenvalues are, set the target value, for
> instance:
> > ./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5
> >
> > Jose
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/66149301/attachment.html>

From vijay.gopal.c at gmail.com  Thu May 14 12:13:52 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Thu, 14 May 2015 19:13:52 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
	<AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
Message-ID: <CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>

oups sorry, I send you a smaller one (540540) this should finish in a few
minutes.

It requires the same makefile and irpf90.a library.
so just replace the old problem.c file with this and it should compile.

Thanks again,
 Vijay

On Thu, May 14, 2015 at 7:04 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> Ok. I was able to build the code, but I need a small problem in order to
> be able to run without having to access a big supercomputer.
> Jose
>
>
> El 14/05/2015, a las 18:00, Vijay Gopal Chilkuri escribi?:
>
> > Ok I've sent a job with the -eps_mpd 50 option and I still get the same
> absurd eigenvalues !
> >
> > You already have the driver .c file (called problem.c I also attach this
> file) I send you the makefile.
> >
> > The subroutine unit_l1_ is a fortran subroutine which gives the number
> of nonzero elements in 22 rows at once. This subroutine can be compiled in
> the following way.
> >
> > Firstly, it uses the IRPF90 library.
> > You can download the library from this github link:
> https://github.com/scemama/irpf90.git
> > the binary needed "irpf90" should be in the bin directory, just update
> the relevant PATH environment variables.
> >
> > Second, get the codes that generate the unit_l1_ subroutine from the
> following repository: https://github.com/v1j4y/slepc_version.git
> > please change to branch slepc_testing !
> > make
> > make irpf90.a
> >
> > should give you the irpf90.a library nessessary to compile the problem.c
> file, I attach the makefile to compile the problem.c file with this mail.
> > To compile the final executable, please do "make problem"
> >
> > and that should be enough !
> > sorry that it is not simpler.
> >
> > Please write if you run into problems.
> >
> > Thanks,
> >  Vijay
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/cebfaa2e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: problemsmall.c
Type: text/x-csrc
Size: 11699 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/cebfaa2e/attachment-0001.c>

From jroman at dsic.upv.es  Thu May 14 12:24:47 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 19:24:47 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
Message-ID: <50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>


El 14/05/2015, a las 19:13, venkatesh g escribi?:

> Hi, 
> 1. I am not calling EPSSetFromOptions in source code. I am using the ex7.c given in the SLEPc webpage. 
> 2. A-matrix is not singular. Only B is singular.
> 3. I will try with MUMPS.
> 4. Matrix sizes are same as in Matlab. If B is not singular it is giving same result as Matlab, but if B is singular it gives the zero pivot issue which Matlab doesnt give. 
> So is there a way ?
> 

The ex7.c example calls EPSSetFromOptions(). If it is not called, then command-line options are ignored.

It is strange that you get zero pivots when factoring A-sigma*B for different values of sigma. Try with MUMPS.

Jose


From venkateshgk.j at gmail.com  Thu May 14 12:29:24 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Thu, 14 May 2015 22:59:24 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
Message-ID: <CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>

I am trying to use ARPACK. will it help?

It is not linking Arpack however, when I give ./configure
--with-arpack-dir=..... command from manual.

Venkatesh

On Thu, May 14, 2015 at 10:54 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 14/05/2015, a las 19:13, venkatesh g escribi?:
>
> > Hi,
> > 1. I am not calling EPSSetFromOptions in source code. I am using the
> ex7.c given in the SLEPc webpage.
> > 2. A-matrix is not singular. Only B is singular.
> > 3. I will try with MUMPS.
> > 4. Matrix sizes are same as in Matlab. If B is not singular it is giving
> same result as Matlab, but if B is singular it gives the zero pivot issue
> which Matlab doesnt give.
> > So is there a way ?
> >
>
> The ex7.c example calls EPSSetFromOptions(). If it is not called, then
> command-line options are ignored.
>
> It is strange that you get zero pivots when factoring A-sigma*B for
> different values of sigma. Try with MUMPS.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/998f5471/attachment.html>

From jroman at dsic.upv.es  Thu May 14 12:31:03 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 19:31:03 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
Message-ID: <B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>

This has nothing to do with ARPACK.


El 14/05/2015, a las 19:29, venkatesh g escribi?:

> I am trying to use ARPACK. will it help?
> 
> It is not linking Arpack however, when I give ./configure --with-arpack-dir=..... command from manual.
> 
> Venkatesh
> 
> On Thu, May 14, 2015 at 10:54 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> El 14/05/2015, a las 19:13, venkatesh g escribi?:
> 
> > Hi,
> > 1. I am not calling EPSSetFromOptions in source code. I am using the ex7.c given in the SLEPc webpage.
> > 2. A-matrix is not singular. Only B is singular.
> > 3. I will try with MUMPS.
> > 4. Matrix sizes are same as in Matlab. If B is not singular it is giving same result as Matlab, but if B is singular it gives the zero pivot issue which Matlab doesnt give.
> > So is there a way ?
> >
> 
> The ex7.c example calls EPSSetFromOptions(). If it is not called, then command-line options are ignored.
> 
> It is strange that you get zero pivots when factoring A-sigma*B for different values of sigma. Try with MUMPS.
> 
> Jose
> 
> 


From venkateshgk.j at gmail.com  Thu May 14 12:34:33 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Thu, 14 May 2015 23:04:33 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
Message-ID: <CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>

Actually krylovschur it is giving excellent results when my B matrix is not
singular.

This time my B matrix is large with very little non-zero entries. There are
entire rows and columns with zeros.
So other than MUMPS, which I will try now, is there another way ?

Venkatesh

On Thu, May 14, 2015 at 11:01 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> This has nothing to do with ARPACK.
>
>
> El 14/05/2015, a las 19:29, venkatesh g escribi?:
>
> > I am trying to use ARPACK. will it help?
> >
> > It is not linking Arpack however, when I give ./configure
> --with-arpack-dir=..... command from manual.
> >
> > Venkatesh
> >
> > On Thu, May 14, 2015 at 10:54 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >
> > El 14/05/2015, a las 19:13, venkatesh g escribi?:
> >
> > > Hi,
> > > 1. I am not calling EPSSetFromOptions in source code. I am using the
> ex7.c given in the SLEPc webpage.
> > > 2. A-matrix is not singular. Only B is singular.
> > > 3. I will try with MUMPS.
> > > 4. Matrix sizes are same as in Matlab. If B is not singular it is
> giving same result as Matlab, but if B is singular it gives the zero pivot
> issue which Matlab doesnt give.
> > > So is there a way ?
> > >
> >
> > The ex7.c example calls EPSSetFromOptions(). If it is not called, then
> command-line options are ignored.
> >
> > It is strange that you get zero pivots when factoring A-sigma*B for
> different values of sigma. Try with MUMPS.
> >
> > Jose
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/85f0bf1a/attachment.html>

From jroman at dsic.upv.es  Thu May 14 12:43:57 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 14 May 2015 19:43:57 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
Message-ID: <4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>


El 14/05/2015, a las 19:34, venkatesh g escribi?:

> Actually krylovschur it is giving excellent results when my B matrix is not singular. 
> 
> This time my B matrix is large with very little non-zero entries. There are entire rows and columns with zeros. 
> So other than MUMPS, which I will try now, is there another way ?
> 
> Venkatesh
> 

This line:

$ ./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5

will factor A-sigma*B for sigma=0.5 if command line arguments are processed correctly. So even if B has many zeros, the factorization should not fail. You could try tweaking PETSc's LU factorization parameters, or use an external LU library such as MUMPS or UMFPACK, to see if that helps. This has nothing to do with the (outer) solver used for the eigenvalues.

Jose


From lb2653 at columbia.edu  Thu May 14 14:17:33 2015
From: lb2653 at columbia.edu (Luc Berger-Vergiat)
Date: Thu, 14 May 2015 15:17:33 -0400
Subject: [petsc-users] Setting Pmat for an ASM subksp
Message-ID: <5554F4CD.1010708@columbi.edu>

Hi all,
I am using the ASM preconditioner and it works fine with regular options.
However I am having difficulties setting the Pmat operator for on of my 
ASM domain.
Here is what I coded:

          call PCASMSetLocalSubdomains(pcdd,2,isddi,isddi,ierr)
          call PetscOptionsGetInt(PETSC_NULL_CHARACTER,
      &        '-n_overlap',n_ol,chk,ierr)
          if (chk.eqv.PETSC_TRUE) then
             call PCASMSetOverlap(pcdd,n_ol,ierr)
          endif
          call PCSetUp(pcdd,ierr)
          call KSPSetUp(schurKSP,ierr)
          call PCASMGetSubKSP(pcdd,nddl,fbl,ddsubksp, ierr)

          do i = 1,nddl
             if(i.eq.1) then
                if( (nstep.eq.1).and.(niter.eq.0) ) then
                   call KSPGetOperators(ddsubksp(i),HKmat,
      &                 HPmat,ierr)
                else
                   call KSPGetOperators(ddsubksp(i),HKmat,
      &                 PETSC_NULL_OBJECT,ierr)
                endif
                call PetscObjectReference(HKmat, ierr)
                call PetscObjectReference(HPmat, ierr)
                call KSPSetOperators(ddsubksp(i),HKmat,HPmat,ierr)
                call PetscObjectDereference(HKmat, ierr)
                call PetscObjectDereference(HPmat, ierr)
             endif
             call KSPGetPC(ddsubksp(i),ddsubpc(i),ierr)
             call PCSetType(ddsubpc(i),PCLU,ierr)
             call KSPSetType(ddsubksp(i),KSPPREONLY,ierr)
          enddo

I am suspecting that the PetscObjectReference is not working well based 
on the following output from valgrind:

==24001== Invalid read of size 4
==24001==    at 0x8EDC69: PetscObjectReference (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x8F5778: petscobjectreference_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4331A2: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD9BAE3: PCApply_Shell (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD74AC3: PCApply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0FE2D: KSPSolve_PREONLY (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFD7E3: KSPSolve (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41BEC: kspsolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4288ED: usolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x460179: psolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==  Address 0x802ea30 is 64 bytes inside a block of size 1,072 free'd
==24001==    at 0x4C2BDEC: free (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24001==    by 0xA66C4C: PetscFreeAlign (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x95A5F6: MatDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x9672BC: MatDestroyMatrices (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDDD221: PCReset_ASM (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73A26: PCReset (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73BB7: PCDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFFF53: KSPDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41C1F: kspdestroy_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x432F00: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==
==24001== Invalid read of size 4
==24001==    at 0x8EDC69: PetscObjectReference (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD76924: PCSetOperators (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE045DD: KSPSetOperators (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41B40: kspsetoperators_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4331C4: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD9BAE3: PCApply_Shell (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD74AC3: PCApply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0FE2D: KSPSolve_PREONLY (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFD7E3: KSPSolve (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41BEC: kspsolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==  Address 0x802ea30 is 64 bytes inside a block of size 1,072 free'd
==24001==    at 0x4C2BDEC: free (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24001==    by 0xA66C4C: PetscFreeAlign (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x95A5F6: MatDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x9672BC: MatDestroyMatrices (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDDD221: PCReset_ASM (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73A26: PCReset (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73BB7: PCDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFFF53: KSPDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41C1F: kspdestroy_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x432F00: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==
==24001== Invalid read of size 8
==24001==    at 0x96999F: MatGetNullSpace (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0468C: KSPSetOperators (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41B40: kspsetoperators_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4331C4: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD9BAE3: PCApply_Shell (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD74AC3: PCApply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0FE2D: KSPSolve_PREONLY (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFD7E3: KSPSolve (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41BEC: kspsolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4288ED: usolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==  Address 0x802ed98 is 936 bytes inside a block of size 1,072 
free'd
==24001==    at 0x4C2BDEC: free (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24001==    by 0xA66C4C: PetscFreeAlign (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x95A5F6: MatDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x9672BC: MatDestroyMatrices (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDDD221: PCReset_ASM (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73A26: PCReset (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73BB7: PCDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFFF53: KSPDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41C1F: kspdestroy_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x432F00: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==
==24001== Invalid read of size 8
==24001==    at 0x8EDC90: PetscObjectDereference (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x8F5798: petscobjectdereference_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4331EB: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD9BAE3: PCApply_Shell (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD74AC3: PCApply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0FE2D: KSPSolve_PREONLY (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFD7E3: KSPSolve (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41BEC: kspsolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4288ED: usolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x460179: psolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==  Address 0x802e9f8 is 8 bytes inside a block of size 1,072 free'd
==24001==    at 0x4C2BDEC: free (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24001==    by 0xA66C4C: PetscFreeAlign (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x95A5F6: MatDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x9672BC: MatDestroyMatrices (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDDD221: PCReset_ASM (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73A26: PCReset (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD73BB7: PCDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFFF53: KSPDestroy (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41C1F: kspdestroy_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x432F00: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==
==24001== Invalid read of size 8
==24001==    at 0x8EDC94: PetscObjectDereference (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x8F5798: petscobjectdereference_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4331EB: reuseschur_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4297F5: feapuserpcapply_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA3FB56: ourshellapply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD9BAE3: PCApply_Shell (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xD74AC3: PCApply (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xE0FE2D: KSPSolve_PREONLY (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xDFD7E3: KSPSolve (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0xA41BEC: kspsolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x4288ED: usolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==    by 0x460179: psolve_ (in 
/home/luc/research/feap_repo/ShearBands/parfeap/feap)
==24001==  Address 0x10 is not stack'd, malloc'd or (recently) free'd
==24001==

Can you let me know how I should issue the PetscObejctReference 
subroutine call?
If PetscObejctReference is not fully working in FORTRAN (that's always a 
possibility), I assume that I cannot do the following:

       call MatGetSubMatrix(Kmat, isddi(1), isddi(1), 
MAT_INITIAL_MATRIX, DDPMat, ierr)
       call KSPSetOperators(ddsubksp(i),DDPMat,DDPMat,ierr)

because of the overlap that is set after I computed issddi(1), is there 
a way to get an updated isddi(1) that contain the overlap created by 
PCASMSetOverlap?

-- 
Best,
Luc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/5219696c/attachment-0001.html>

From jed at jedbrown.org  Thu May 14 15:56:02 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 14 May 2015 14:56:02 -0600
Subject: [petsc-users] Setting Pmat for an ASM subksp
In-Reply-To: <5554F4CD.1010708@columbi.edu>
References: <5554F4CD.1010708@columbi.edu>
Message-ID: <87oalm6fq5.fsf@jedbrown.org>

Luc Berger-Vergiat <lb2653 at columbia.edu> writes:

> Hi all,
> I am using the ASM preconditioner and it works fine with regular options.
> However I am having difficulties setting the Pmat operator for on of my 
> ASM domain.
> Here is what I coded:
>
>           call PCASMSetLocalSubdomains(pcdd,2,isddi,isddi,ierr)
>           call PetscOptionsGetInt(PETSC_NULL_CHARACTER,
>       &        '-n_overlap',n_ol,chk,ierr)
>           if (chk.eqv.PETSC_TRUE) then
>              call PCASMSetOverlap(pcdd,n_ol,ierr)
>           endif
>           call PCSetUp(pcdd,ierr)
>           call KSPSetUp(schurKSP,ierr)
>           call PCASMGetSubKSP(pcdd,nddl,fbl,ddsubksp, ierr)

   Fortran note:
   The output argument 'ksp' must be an array of sufficient length or NULL_OBJECT. The latter can be used to learn the necessary length.

Is it allocated and long enough?

>           do i = 1,nddl
>              if(i.eq.1) then
>                 if( (nstep.eq.1).and.(niter.eq.0) ) then
>                    call KSPGetOperators(ddsubksp(i),HKmat,
>       &                 HPmat,ierr)
>                 else
>                    call KSPGetOperators(ddsubksp(i),HKmat,
>       &                 PETSC_NULL_OBJECT,ierr)
>                 endif
>                 call PetscObjectReference(HKmat, ierr)
>                 call PetscObjectReference(HPmat, ierr)
>                 call KSPSetOperators(ddsubksp(i),HKmat,HPmat,ierr)

So you're reusing the Pmat from the first subdomain on each process?
How do you know it's the right size?

>                 call PetscObjectDereference(HKmat, ierr)
>                 call PetscObjectDereference(HPmat, ierr)
>              endif
>              call KSPGetPC(ddsubksp(i),ddsubpc(i),ierr)
>              call PCSetType(ddsubpc(i),PCLU,ierr)
>              call KSPSetType(ddsubksp(i),KSPPREONLY,ierr)
>           enddo
>
> I am suspecting that the PetscObjectReference is not working well based 
> on the following output from valgrind:

Get rid of the PetscObjectReference and PetscObjectDereference
calls. They don't need to be there and could be hiding problems.  Then
send the full Valgrind output.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/e62b1d9b/attachment.pgp>

From gideon.simpson at gmail.com  Thu May 14 17:36:00 2015
From: gideon.simpson at gmail.com (Gideon Simpson)
Date: Thu, 14 May 2015 18:36:00 -0400
Subject: [petsc-users] understanding snes failure
Message-ID: <C00101EA-D100-49E8-A092-858D85C345B0@gmail.com>

Is there a way I can get diagnostic information as to why a snes solve failed?  I?ve noticed that it will run for 1 or 2 iterations, and then cease, and, even with -snes_monitor on, I don?t get any information as to why it terminated.

-gideon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/268f14d5/attachment.html>

From knepley at gmail.com  Thu May 14 18:15:51 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 14 May 2015 18:15:51 -0500
Subject: [petsc-users] understanding snes failure
In-Reply-To: <C00101EA-D100-49E8-A092-858D85C345B0@gmail.com>
References: <C00101EA-D100-49E8-A092-858D85C345B0@gmail.com>
Message-ID: <CAMYG4GnkqkrK+g-fNVSONRG6dA0BJ3Ji+RMr+C-yqteJ7UvdKQ@mail.gmail.com>

-snes_converged_reason

   Matt

On Thu, May 14, 2015 at 5:36 PM, Gideon Simpson <gideon.simpson at gmail.com>
wrote:

> Is there a way I can get diagnostic information as to why a snes solve
> failed?  I?ve noticed that it will run for 1 or 2 iterations, and then
> cease, and, even with -snes_monitor on, I don?t get any information as to
> why it terminated.
>
> -gideon
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150514/6c8232fc/attachment.html>

From bsmith at mcs.anl.gov  Thu May 14 18:41:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 14 May 2015 18:41:28 -0500
Subject: [petsc-users] understanding snes failure
In-Reply-To: <CAMYG4GnkqkrK+g-fNVSONRG6dA0BJ3Ji+RMr+C-yqteJ7UvdKQ@mail.gmail.com>
References: <C00101EA-D100-49E8-A092-858D85C345B0@gmail.com>
	<CAMYG4GnkqkrK+g-fNVSONRG6dA0BJ3Ji+RMr+C-yqteJ7UvdKQ@mail.gmail.com>
Message-ID: <BFDC66AF-F1E8-4AA2-B094-E8C3F4C67B86@mcs.anl.gov>


  Also -snes_linesearch_monitor  can be useful; sometime -ksp_monitor_true_residual and -ksp_converged_reason are useful also


> On May 14, 2015, at 6:15 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> -snes_converged_reason
> 
>    Matt
> 
> On Thu, May 14, 2015 at 5:36 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> Is there a way I can get diagnostic information as to why a snes solve failed?  I?ve noticed that it will run for 1 or 2 iterations, and then cease, and, even with -snes_monitor on, I don?t get any information as to why it terminated.
> 
> -gideon
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From extendnqvw at gmail.com  Fri May 15 03:59:20 2015
From: extendnqvw at gmail.com (extendnqvw at gmail.com)
Date: Fri, 15 May 2015 08:59:20 +0000
Subject: [petsc-users] =?gb2312?b?z9bU2s3iw7PTprjD1ve2r6Gjy9HL99L9x+bHv7Tz?=
	=?gb2312?b?u7nKx0IyQg==?=
Message-ID: <089e0160a59c53fb8f05161b0f74@google.com>

????????????????
????????????????????????????????????
??????????????????????
?????????????????
????QQ2651061613 ??10???????




????????? ?????????????????B2B? ?????
??????
https://docs.google.com/forms/d/1b7fHQmUzVe73Y1G7pKGAPA2QHBjE9QNEI7MPJljuFTs/viewform?c=0&w=1&usp=mail_form_link
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/7b45182d/attachment.html>

From venkateshgk.j at gmail.com  Fri May 15 05:34:14 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 15 May 2015 16:04:14 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
Message-ID: <CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>

Hi,
I configured and installed MUMPS along with PETSc and installed SLEPc.

Now I am trying to make the executable for ex7.c from SLEPc examples.

I am getting following errors while doing make:

../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
In function `MatDestroy_MUMPS':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:493:
undefined reference to `zmumps_c'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
In function `MatSolve_MUMPS':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:546:
undefined reference to `zmumps_c'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
In function `MatFactorNumeric_MUMPS':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:664:
undefined reference to `zmumps_c'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
In function `PetscInitializeMUMPS':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:799:
undefined reference to `zmumps_c'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
In function `MatLUFactorSymbolic_AIJMUMPS':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:896:
undefined reference to `zmumps_c'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:974:
more undefined references to `zmumps_c' follow
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o):
In function `MatPartitioningApply_Parmetis':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:101:
undefined reference to `ParMETIS_V3_PartKway'
../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o):
In function `MatMeshToCellGraph':
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:360:
undefined reference to `ParMETIS_V3_Mesh2Dual'
collect2: ld returned 1 exit status
make: *** [ex7] Error 1

Pls let me know what to do ?

Venkatesh




On Thu, May 14, 2015 at 11:13 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 14/05/2015, a las 19:34, venkatesh g escribi?:
>
> > Actually krylovschur it is giving excellent results when my B matrix is
> not singular.
> >
> > This time my B matrix is large with very little non-zero entries. There
> are entire rows and columns with zeros.
> > So other than MUMPS, which I will try now, is there another way ?
> >
> > Venkatesh
> >
>
> This line:
>
> $ ./ex7 -f1 A1 -f2 B1 -st_type sinvert -eps_target 0.5
>
> will factor A-sigma*B for sigma=0.5 if command line arguments are
> processed correctly. So even if B has many zeros, the factorization should
> not fail. You could try tweaking PETSc's LU factorization parameters, or
> use an external LU library such as MUMPS or UMFPACK, to see if that helps.
> This has nothing to do with the (outer) solver used for the eigenvalues.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/3d94042b/attachment-0001.html>

From jroman at dsic.upv.es  Fri May 15 05:42:40 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 15 May 2015 12:42:40 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
Message-ID: <9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>


El 15/05/2015, a las 12:34, venkatesh g escribi?:

> Hi, 
> I configured and installed MUMPS along with PETSc and installed SLEPc. 
> 
> Now I am trying to make the executable for ex7.c from SLEPc examples.
> 
> I am getting following errors while doing make: 
> 
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o): In function `MatDestroy_MUMPS':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:493: undefined reference to `zmumps_c'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o): In function `MatSolve_MUMPS':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:546: undefined reference to `zmumps_c'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o): In function `MatFactorNumeric_MUMPS':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:664: undefined reference to `zmumps_c'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o): In function `PetscInitializeMUMPS':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:799: undefined reference to `zmumps_c'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o): In function `MatLUFactorSymbolic_AIJMUMPS':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:896: undefined reference to `zmumps_c'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:974: more undefined references to `zmumps_c' follow
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o): In function `MatPartitioningApply_Parmetis':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:101: undefined reference to `ParMETIS_V3_PartKway'
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o): In function `MatMeshToCellGraph':
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:360: undefined reference to `ParMETIS_V3_Mesh2Dual'
> collect2: ld returned 1 exit status
> make: *** [ex7] Error 1
> 
> Pls let me know what to do ?
> 
> Venkatesh
> 


Suggest upgrading to the latest PETSc/SLEPc, version 3.3 is almost 3-years old.

Did 'make test' on $PETSC_DIR work?

Did you install a standalone MUMPS yourself? Maybe it is better to let PETSc install it for you. In 3.5, run PETSc's configure with:  --download-mumps --download-metis --download-parmetis --download-scalapack

Jose


From venkateshgk.j at gmail.com  Fri May 15 05:49:41 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 15 May 2015 16:19:41 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
Message-ID: <CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>

Hi,
'make test' was successful on the $PETSC_DIR.
I configured MUMPS, metis, parmetis, scalapack through
'--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..

I will anyway download the new version and start installing it now., Is it
due to some flag issue ? I have attached my makefile.

Venkatesh




On Fri, May 15, 2015 at 4:12 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 15/05/2015, a las 12:34, venkatesh g escribi?:
>
> > Hi,
> > I configured and installed MUMPS along with PETSc and installed SLEPc.
> >
> > Now I am trying to make the executable for ex7.c from SLEPc examples.
> >
> > I am getting following errors while doing make:
> >
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
> In function `MatDestroy_MUMPS':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:493:
> undefined reference to `zmumps_c'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
> In function `MatSolve_MUMPS':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:546:
> undefined reference to `zmumps_c'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
> In function `MatFactorNumeric_MUMPS':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:664:
> undefined reference to `zmumps_c'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
> In function `PetscInitializeMUMPS':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:799:
> undefined reference to `zmumps_c'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):
> In function `MatLUFactorSymbolic_AIJMUMPS':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:896:
> undefined reference to `zmumps_c'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(mumps.c.o):/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c:974:
> more undefined references to `zmumps_c' follow
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o):
> In function `MatPartitioningApply_Parmetis':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:101:
> undefined reference to `ParMETIS_V3_PartKway'
> >
> ../../petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib//libpetsc.a(pmetis.c.o):
> In function `MatMeshToCellGraph':
> >
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/partition/impls/pmetis/pmetis.c:360:
> undefined reference to `ParMETIS_V3_Mesh2Dual'
> > collect2: ld returned 1 exit status
> > make: *** [ex7] Error 1
> >
> > Pls let me know what to do ?
> >
> > Venkatesh
> >
>
>
> Suggest upgrading to the latest PETSc/SLEPc, version 3.3 is almost 3-years
> old.
>
> Did 'make test' on $PETSC_DIR work?
>
> Did you install a standalone MUMPS yourself? Maybe it is better to let
> PETSc install it for you. In 3.5, run PETSc's configure with:
> --download-mumps --download-metis --download-parmetis --download-scalapack
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/fd1f38b0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Makefile
Type: application/octet-stream
Size: 792 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/fd1f38b0/attachment.obj>

From jroman at dsic.upv.es  Fri May 15 05:52:28 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 15 May 2015 12:52:28 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
Message-ID: <618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>


El 15/05/2015, a las 12:49, venkatesh g escribi?:

> Hi, 
> 'make test' was successful on the $PETSC_DIR. 
> I configured MUMPS, metis, parmetis, scalapack through '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc.. 
> 
> I will anyway download the new version and start installing it now., Is it due to some flag issue ? I have attached my makefile. 
> 
> Venkatesh
> 

You must link with all libraries, otherwise you will get undefined symbols. The easiest way is to include PETSc and SLEPc makefile definitions. See for instance section 1.4.2 of SLEPc's users manual.

Jose


From venkateshgk.j at gmail.com  Fri May 15 06:48:39 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 15 May 2015 17:18:39 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
	<618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
Message-ID: <CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>

Ok, I followed this process and compiled ex7 successfully.

Now I give the command: './ex7 -f1 A11 -f2 B11 -eps_nev 4 -eps_target 0.5
-st_type sinvert'

The error is the zero pivot problem:

 Reading COMPLEX matrices from binary files...
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Detected zero pivot in LU factorization:
see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
[0]PETSC ERROR: Zero pivot row 420 value 0 tolerance 2.22045e-14!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
CDT 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
venkatesh Fri May 15 17:07:02 2015
[0]PETSC ERROR: Libraries linked from
/cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
[0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
--with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
--download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
--download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
--download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
--download-blacs=/cluster/share/apps/blacs-dev.tar.gz
--download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
--download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
--download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
--with-scalar-type=complex --with-fortran-kernels=generic
--with-large-file-io=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatPivotCheck_none() line 583 in
/cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
[0]PETSC ERROR: MatPivotCheck() line 602 in
/cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
[0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 570 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/seq/aijfact.c
[0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCSetUp_LU() line 160 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: PCSetUp() line 832 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: PCSetUp_Redundant() line 176 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/redundant/redundant.c
[0]PETSC ERROR: PCSetUp() line 832 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: STSetUp_Sinvert() line 153 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: STSetUp() line 280 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
[0]PETSC ERROR: EPSSetUp() line 204 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
[0]PETSC ERROR: EPSSolve() line 109 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
[0]PETSC ERROR: main() line 148 in ex7.c
application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0

Venkatesh

On Fri, May 15, 2015 at 4:22 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 15/05/2015, a las 12:49, venkatesh g escribi?:
>
> > Hi,
> > 'make test' was successful on the $PETSC_DIR.
> > I configured MUMPS, metis, parmetis, scalapack through
> '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..
> >
> > I will anyway download the new version and start installing it now., Is
> it due to some flag issue ? I have attached my makefile.
> >
> > Venkatesh
> >
>
> You must link with all libraries, otherwise you will get undefined
> symbols. The easiest way is to include PETSc and SLEPc makefile
> definitions. See for instance section 1.4.2 of SLEPc's users manual.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/24ecff0b/attachment-0001.html>

From jroman at dsic.upv.es  Fri May 15 07:44:01 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 15 May 2015 14:44:01 +0200
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
	<618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
	<CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>
Message-ID: <ADA696F0-6CD3-48EC-89C5-3BE5BDD06F70@dsic.upv.es>

You have to choose MUMPS as the solver. ONCE AGAIN, read section 3.4.1 of the users manual.
Jose


El 15/05/2015, a las 13:48, venkatesh g escribi?:

> Ok, I followed this process and compiled ex7 successfully. 
> 
> Now I give the command: './ex7 -f1 A11 -f2 B11 -eps_nev 4 -eps_target 0.5 -st_type sinvert'
> 
> The error is the zero pivot problem:
> 
>  Reading COMPLEX matrices from binary files...
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Detected zero pivot in LU factorization:
> see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
> [0]PETSC ERROR: Zero pivot row 420 value 0 tolerance 2.22045e-14!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by venkatesh Fri May 15 17:07:02 2015
> [0]PETSC ERROR: Libraries linked from /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
> [0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz --with-scalar-type=complex --with-fortran-kernels=generic --with-large-file-io=1
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: MatPivotCheck_none() line 583 in /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
> [0]PETSC ERROR: MatPivotCheck() line 602 in /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
> [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 570 in /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/seq/aijfact.c
> [0]PETSC ERROR: MatLUFactorNumeric() line 2803 in /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
> [0]PETSC ERROR: PCSetUp_LU() line 160 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
> [0]PETSC ERROR: PCSetUp() line 832 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: PCSetUp_Redundant() line 176 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/redundant/redundant.c
> [0]PETSC ERROR: PCSetUp() line 832 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: STSetUp_Sinvert() line 153 in /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
> [0]PETSC ERROR: STSetUp() line 280 in /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
> [0]PETSC ERROR: EPSSetUp() line 204 in /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
> [0]PETSC ERROR: EPSSolve() line 109 in /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
> [0]PETSC ERROR: main() line 148 in ex7.c
> application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> 
> Venkatesh
> 
> On Fri, May 15, 2015 at 4:22 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> El 15/05/2015, a las 12:49, venkatesh g escribi?:
> 
> > Hi,
> > 'make test' was successful on the $PETSC_DIR.
> > I configured MUMPS, metis, parmetis, scalapack through '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..
> >
> > I will anyway download the new version and start installing it now., Is it due to some flag issue ? I have attached my makefile.
> >
> > Venkatesh
> >
> 
> You must link with all libraries, otherwise you will get undefined symbols. The easiest way is to include PETSc and SLEPc makefile definitions. See for instance section 1.4.2 of SLEPc's users manual.
> 
> Jose
> 
> 


From jroman at dsic.upv.es  Fri May 15 07:55:51 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 15 May 2015 14:55:51 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
	<AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
	<CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>
Message-ID: <C1344D64-89F3-4F71-B4F7-D1E1204717CC@dsic.upv.es>


El 14/05/2015, a las 19:13, Vijay Gopal Chilkuri escribi?:

> oups sorry, I send you a smaller one (540540) this should finish in a few minutes.
> 
> It requires the same makefile and irpf90.a library.
> so just replace the old problem.c file with this and it should compile.
> 
> Thanks again,
>  Vijay
> 

I was able to compute 300 eigenvalues of this matrix of size 540540. All eigenvalues are in the range -4.70811 .. -4.613807, and the associated residual is always below 1e-9.

There must be something in your software configuration that is causing problems. I would suggest trying with a basic PETSc/SLEPc configuration, with no openmp flags, using --download-fblaslapack (instead of MKL). Also, although it should not make any difference, you may want to try with a smaller number of MPI processes (rather than 741).

Jose


From vijay.gopal.c at gmail.com  Fri May 15 08:00:04 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Fri, 15 May 2015 15:00:04 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <C1344D64-89F3-4F71-B4F7-D1E1204717CC@dsic.upv.es>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
	<AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
	<CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>
	<C1344D64-89F3-4F71-B4F7-D1E1204717CC@dsic.upv.es>
Message-ID: <CAJa9Fi9-aEa60SFNuRe+kPF5=zrm5pYvFzBKUbYyBtMV-qyuLQ@mail.gmail.com>

Yes, those seem to be the right eigenvalues.
Ok so the solution is to recompile PETSc/SLEPc with a basic configuration
and test with --with-debugging=1

Would it make a difference if I use *-esp_type lanczos *or some other
diagonalization procedure ?

I'll run the test with a new version of PETSc/SLEPc and report back.

Thanks a  lot,
 Vijay

On Fri, May 15, 2015 at 2:55 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 14/05/2015, a las 19:13, Vijay Gopal Chilkuri escribi?:
>
> > oups sorry, I send you a smaller one (540540) this should finish in a
> few minutes.
> >
> > It requires the same makefile and irpf90.a library.
> > so just replace the old problem.c file with this and it should compile.
> >
> > Thanks again,
> >  Vijay
> >
>
> I was able to compute 300 eigenvalues of this matrix of size 540540. All
> eigenvalues are in the range -4.70811 .. -4.613807, and the associated
> residual is always below 1e-9.
>
> There must be something in your software configuration that is causing
> problems. I would suggest trying with a basic PETSc/SLEPc configuration,
> with no openmp flags, using --download-fblaslapack (instead of MKL). Also,
> although it should not make any difference, you may want to try with a
> smaller number of MPI processes (rather than 741).
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/72741c4d/attachment.html>

From jroman at dsic.upv.es  Fri May 15 08:18:27 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 15 May 2015 15:18:27 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <CAJa9Fi9-aEa60SFNuRe+kPF5=zrm5pYvFzBKUbYyBtMV-qyuLQ@mail.gmail.com>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
	<AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
	<CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>
	<C1344D64-89F3-4F71-B4F7-D1E1204717CC@dsic.upv.es>
	<CAJa9Fi9-aEa60SFNuRe+kPF5=zrm5pYvFzBKUbYyBtMV-qyuLQ@mail.gmail.com>
Message-ID: <37D00A8A-CD9B-4FD9-8A62-E42F4A89889D@dsic.upv.es>


El 15/05/2015, a las 15:00, Vijay Gopal Chilkuri escribi?:

> Yes, those seem to be the right eigenvalues.
> Ok so the solution is to recompile PETSc/SLEPc with a basic configuration and test with --with-debugging=1 

Yes, it is always good to work --with-debugging=1 until you know your code is working correctly. But anyway, I did the computation --with-debugging=0, with gnu compilers. It is more important to avoid multiple threads, for the moment, both in the LAPACK linked with SLEPc (avoid MKL), and the code that generates the matrix (just in case there was a problem in computing the coefficients).

> 
> Would it make a difference if I use -esp_type lanczos or some other diagonalization procedure ?

Do not use lanczos, it is just a worse algorithm than the default solver (explicit vs implicit restart).

In principle, methods of the CG type are appropriate for your problem, but convergence may be too slow. You can try RQCG with icc preconditioner. I am currently implementing LOBPCG in the development version - I will try with your matrices.

Jose


> 
> I'll run the test with a new version of PETSc/SLEPc and report back.
> 
> Thanks a  lot,
>  Vijay
> 
> On Fri, May 15, 2015 at 2:55 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> El 14/05/2015, a las 19:13, Vijay Gopal Chilkuri escribi?:
> 
> > oups sorry, I send you a smaller one (540540) this should finish in a few minutes.
> >
> > It requires the same makefile and irpf90.a library.
> > so just replace the old problem.c file with this and it should compile.
> >
> > Thanks again,
> >  Vijay
> >
> 
> I was able to compute 300 eigenvalues of this matrix of size 540540. All eigenvalues are in the range -4.70811 .. -4.613807, and the associated residual is always below 1e-9.
> 
> There must be something in your software configuration that is causing problems. I would suggest trying with a basic PETSc/SLEPc configuration, with no openmp flags, using --download-fblaslapack (instead of MKL). Also, although it should not make any difference, you may want to try with a smaller number of MPI processes (rather than 741).
> 
> Jose
> 
> 


From vijay.gopal.c at gmail.com  Fri May 15 08:21:27 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Fri, 15 May 2015 15:21:27 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
	[SOLVED]
Message-ID: <CAJa9Fi-aqUyzeAb5-+=6iRMTyLir2o-A64Bd=GW1ZqeQQ_5o5A@mail.gmail.com>

To close this thread let me point out the problem.

The problem occured due to the distribution of the number of processors
over the total number of nodes used (as Jose E. Roman pointed out.)
The problem was solved and all the 300 (good) eigenvalues, properly
converged,
were obtained when the correct distribution of the number of processors
over the nodes was ascertained using *numactl *.

TLDR, incorrect distribution of processors using numactl may lead to bogus
eigenvalues.

Thanks Jose E. Roman !

Problem is now solved.

With regards,
 Vijay

On Fri, May 15, 2015 at 3:00 PM, Vijay Gopal Chilkuri <
vijay.gopal.c at gmail.com> wrote:

> Yes, those seem to be the right eigenvalues.
> Ok so the solution is to recompile PETSc/SLEPc with a basic configuration
> and test with --with-debugging=1
>
> Would it make a difference if I use *-esp_type lanczos *or some other
> diagonalization procedure ?
>
> I'll run the test with a new version of PETSc/SLEPc and report back.
>
> Thanks a  lot,
>  Vijay
>
> On Fri, May 15, 2015 at 2:55 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>
>>
>> El 14/05/2015, a las 19:13, Vijay Gopal Chilkuri escribi?:
>>
>> > oups sorry, I send you a smaller one (540540) this should finish in a
>> few minutes.
>> >
>> > It requires the same makefile and irpf90.a library.
>> > so just replace the old problem.c file with this and it should compile.
>> >
>> > Thanks again,
>> >  Vijay
>> >
>>
>> I was able to compute 300 eigenvalues of this matrix of size 540540. All
>> eigenvalues are in the range -4.70811 .. -4.613807, and the associated
>> residual is always below 1e-9.
>>
>> There must be something in your software configuration that is causing
>> problems. I would suggest trying with a basic PETSc/SLEPc configuration,
>> with no openmp flags, using --download-fblaslapack (instead of MKL). Also,
>> although it should not make any difference, you may want to try with a
>> smaller number of MPI processes (rather than 741).
>>
>> Jose
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/ca49aecc/attachment.html>

From vijay.gopal.c at gmail.com  Fri May 15 08:26:12 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Fri, 15 May 2015 15:26:12 +0200
Subject: [petsc-users] SLEPc Bogus eigenvalues for large -eps_nev
In-Reply-To: <37D00A8A-CD9B-4FD9-8A62-E42F4A89889D@dsic.upv.es>
References: <CAJa9Fi9WAawxXJg9v-VD_N2kqS8twGLumLNS02P+G1XmZwfjwQ@mail.gmail.com>
	<BC9DA025-945D-485A-8869-0901BC3E8602@dsic.upv.es>
	<CAJa9Fi9P7abvxuzdbYWgeMNE05xCY=nB5R0Up=Njr4Z1Hgo6-A@mail.gmail.com>
	<DFAC1580-CFAD-4F34-ADA6-ED79F0D519F5@dsic.upv.es>
	<CAJa9Fi_r4PZuQ8tO_Tu0nS4HM+-udX3_HrxdA-Yjodr54HL0NA@mail.gmail.com>
	<AB636763-97E0-405D-8FD5-D7E7F0CDCBF0@dsic.upv.es>
	<CAJa9Fi_G1U1pfxeN4tOURQqQUvKgaOja3A07_SrTv1RKEnX3KA@mail.gmail.com>
	<C1344D64-89F3-4F71-B4F7-D1E1204717CC@dsic.upv.es>
	<CAJa9Fi9-aEa60SFNuRe+kPF5=zrm5pYvFzBKUbYyBtMV-qyuLQ@mail.gmail.com>
	<37D00A8A-CD9B-4FD9-8A62-E42F4A89889D@dsic.upv.es>
Message-ID: <CAJa9Fi9WqMhqtdP+r+Z6h8dzB1g65w-o5k3beVDWgrx1wtneYQ@mail.gmail.com>

On Fri, May 15, 2015 at 3:18 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

>
> El 15/05/2015, a las 15:00, Vijay Gopal Chilkuri escribi?:
>
> > Yes, those seem to be the right eigenvalues.
> > Ok so the solution is to recompile PETSc/SLEPc with a basic
> configuration and test with --with-debugging=1
>
> Yes, it is always good to work --with-debugging=1 until you know your code
> is working correctly. But anyway, I did the computation --with-debugging=0,
> with gnu compilers. It is more important to avoid multiple threads, for the
> moment, both in the LAPACK linked with SLEPc (avoid MKL), and the code that
> generates the matrix (just in case there was a problem in computing the
> coefficients).
>
>
> > Would it make a difference if I use -esp_type lanczos or some other
> diagonalization procedure ?
>
> Do not use lanczos, it is just a worse algorithm than the default solver
> (explicit vs implicit restart).
>
> In principle, methods of the CG type are appropriate for your problem, but
> convergence may be too slow. You can try RQCG with icc preconditioner. I am
> currently implementing LOBPCG in the development version - I will try with
> your matrices.
>
>
Yes ! that would be great !

At the moment, I'm having a lot of trouble with the assembly time for large
matrices, although the actual diagonalization with
Krylov-Schur takes considerable time too.
Please let me know if you could improve the matrix assembly performance for
large matrices.

Although I have to say that the implicit restart in Krylov-Schur seems to
be working great for my type of matrices.
Please keep in touch if you find something interesting.

Thanks again,
 Vijay


Jose
>
>
> >
> > I'll run the test with a new version of PETSc/SLEPc and report back.
> >
> > Thanks a  lot,
> >  Vijay
> >
> > On Fri, May 15, 2015 at 2:55 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >
> > El 14/05/2015, a las 19:13, Vijay Gopal Chilkuri escribi?:
> >
> > > oups sorry, I send you a smaller one (540540) this should finish in a
> few minutes.
> > >
> > > It requires the same makefile and irpf90.a library.
> > > so just replace the old problem.c file with this and it should compile.
> > >
> > > Thanks again,
> > >  Vijay
> > >
> >
> > I was able to compute 300 eigenvalues of this matrix of size 540540. All
> eigenvalues are in the range -4.70811 .. -4.613807, and the associated
> residual is always below 1e-9.
> >
> > There must be something in your software configuration that is causing
> problems. I would suggest trying with a basic PETSc/SLEPc configuration,
> with no openmp flags, using --download-fblaslapack (instead of MKL). Also,
> although it should not make any difference, you may want to try with a
> smaller number of MPI processes (rather than 741).
> >
> > Jose
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/660218d9/attachment-0001.html>

From venkateshgk.j at gmail.com  Fri May 15 09:54:39 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 15 May 2015 20:24:39 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <ADA696F0-6CD3-48EC-89C5-3BE5BDD06F70@dsic.upv.es>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
	<618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
	<CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>
	<ADA696F0-6CD3-48EC-89C5-3BE5BDD06F70@dsic.upv.es>
Message-ID: <CABzT7hP6uhODEgD3PH=FpF8pCxkXNTCokEZnxvFN+MoVU18_PQ@mail.gmail.com>

Ah. I made mistake by looking at the old manual. I am now reading the
latest version of the manual.

I will do this and get back.

Venkatesh

On Fri, May 15, 2015 at 6:14 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> You have to choose MUMPS as the solver. ONCE AGAIN, read section 3.4.1 of
> the users manual.
> Jose
>
>
> El 15/05/2015, a las 13:48, venkatesh g escribi?:
>
> > Ok, I followed this process and compiled ex7 successfully.
> >
> > Now I give the command: './ex7 -f1 A11 -f2 B11 -eps_nev 4 -eps_target
> 0.5 -st_type sinvert'
> >
> > The error is the zero pivot problem:
> >
> >  Reading COMPLEX matrices from binary files...
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Detected zero pivot in LU factorization:
> > see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
> > [0]PETSC ERROR: Zero pivot row 420 value 0 tolerance 2.22045e-14!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
> 22:15:24 CDT 2013
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
> venkatesh Fri May 15 17:07:02 2015
> > [0]PETSC ERROR: Libraries linked from
> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
> > [0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
> --with-scalar-type=complex --with-fortran-kernels=generic
> --with-large-file-io=1
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: MatPivotCheck_none() line 583 in
> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
> > [0]PETSC ERROR: MatPivotCheck() line 602 in
> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
> > [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 570 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/seq/aijfact.c
> > [0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
> > [0]PETSC ERROR: PCSetUp_LU() line 160 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
> > [0]PETSC ERROR: PCSetUp() line 832 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: KSPSetUp() line 278 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: PCSetUp_Redundant() line 176 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/redundant/redundant.c
> > [0]PETSC ERROR: PCSetUp() line 832 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: KSPSetUp() line 278 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: STSetUp_Sinvert() line 153 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
> > [0]PETSC ERROR: STSetUp() line 280 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
> > [0]PETSC ERROR: EPSSetUp() line 204 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
> > [0]PETSC ERROR: EPSSolve() line 109 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
> > [0]PETSC ERROR: main() line 148 in ex7.c
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> > [unset]: aborting job:
> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
> >
> > Venkatesh
> >
> > On Fri, May 15, 2015 at 4:22 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >
> > El 15/05/2015, a las 12:49, venkatesh g escribi?:
> >
> > > Hi,
> > > 'make test' was successful on the $PETSC_DIR.
> > > I configured MUMPS, metis, parmetis, scalapack through
> '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..
> > >
> > > I will anyway download the new version and start installing it now.,
> Is it due to some flag issue ? I have attached my makefile.
> > >
> > > Venkatesh
> > >
> >
> > You must link with all libraries, otherwise you will get undefined
> symbols. The easiest way is to include PETSc and SLEPc makefile
> definitions. See for instance section 1.4.2 of SLEPc's users manual.
> >
> > Jose
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/97262279/attachment.html>

From fande.kong at colorado.edu  Fri May 15 13:45:47 2015
From: fande.kong at colorado.edu (Fande Kong)
Date: Fri, 15 May 2015 12:45:47 -0600
Subject: [petsc-users] why we have a zero residual?
Message-ID: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>

Hi all,

I was wondering why we will get a zero residual when solving the linear
step of a nonlinear problem? I am solving a fluid-structure interaction
problems using inexact Newton method, within which fGMRES is used together
with a multilevel Schwarz preconditioner. The algorithm works well when the
mesh is small, but it catched a zero residual issue when we turn to use a
little bit large mesh.  The output is listed as follows:

0 SNES Function norm 1.291775443677e-02
  Jacobian: rows 5575441  Column 5575441
call DMGetInterpolation_SpmcsDMMesh
  Jacobian: rows 1298448  Column 1298448
    0 KSP Residual norm 1.291775443677e-02
    1 KSP Residual norm 0.000000000000e+00
Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0


Any suggestions? Thanks.

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/fa0af1c2/attachment.html>

From knepley at gmail.com  Fri May 15 13:49:44 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 15 May 2015 13:49:44 -0500
Subject: [petsc-users] why we have a zero residual?
In-Reply-To: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>
References: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>
Message-ID: <CAMYG4G=YhqmrsqsFzTfPk-=HLUn7cDxqdWEgGb3Kis00f2FRbw@mail.gmail.com>

On Fri, May 15, 2015 at 1:45 PM, Fande Kong <fande.kong at colorado.edu> wrote:

> Hi all,
>
> I was wondering why we will get a zero residual when solving the linear
> step of a nonlinear problem? I am solving a fluid-structure interaction
> problems using inexact Newton method, within which fGMRES is used together
> with a multilevel Schwarz preconditioner. The algorithm works well when the
> mesh is small, but it catched a zero residual issue when we turn to use a
> little bit large mesh.  The output is listed as follows:
>
> 0 SNES Function norm 1.291775443677e-02
>   Jacobian: rows 5575441  Column 5575441
> call DMGetInterpolation_SpmcsDMMesh
>   Jacobian: rows 1298448  Column 1298448
>     0 KSP Residual norm 1.291775443677e-02
>     1 KSP Residual norm 0.000000000000e+00
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0
>

This looks like a mistake in the Jacobian or preconditioner, since you have
a 0 residual, but
a bad descent direction.

  Matt


> Any suggestions? Thanks.
>
> Fande,
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/2eeb78c2/attachment.html>

From karpeev at mcs.anl.gov  Fri May 15 14:04:26 2015
From: karpeev at mcs.anl.gov (Dmitry Karpeyev)
Date: Fri, 15 May 2015 19:04:26 +0000
Subject: [petsc-users] why we have a zero residual?
In-Reply-To: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>
References: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>
Message-ID: <CA+Y_9U+HOtvR=qDpZ4s79dYeehFj3wCV2zdTf3L0u-wwQ-sdKA@mail.gmail.com>

What does -ksp_monitor_true_residual show?

On Fri, May 15, 2015 at 1:45 PM Fande Kong <fande.kong at colorado.edu> wrote:

> Hi all,
>
> I was wondering why we will get a zero residual when solving the linear
> step of a nonlinear problem? I am solving a fluid-structure interaction
> problems using inexact Newton method, within which fGMRES is used together
> with a multilevel Schwarz preconditioner. The algorithm works well when the
> mesh is small, but it catched a zero residual issue when we turn to use a
> little bit large mesh.  The output is listed as follows:
>
> 0 SNES Function norm 1.291775443677e-02
>   Jacobian: rows 5575441  Column 5575441
> call DMGetInterpolation_SpmcsDMMesh
>   Jacobian: rows 1298448  Column 1298448
>     0 KSP Residual norm 1.291775443677e-02
>     1 KSP Residual norm 0.000000000000e+00
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0
>
>
> Any suggestions? Thanks.
>
> Fande,
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/d007a871/attachment.html>

From bsmith at mcs.anl.gov  Fri May 15 14:14:58 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 15 May 2015 14:14:58 -0500
Subject: [petsc-users] why we have a zero residual?
In-Reply-To: <CA+Y_9U+HOtvR=qDpZ4s79dYeehFj3wCV2zdTf3L0u-wwQ-sdKA@mail.gmail.com>
References: <CAN5Wd-LN5f+vPFrjwAAOSRKyDw-VL9dS5JREDJisHgw_KkvuTg@mail.gmail.com>
	<CA+Y_9U+HOtvR=qDpZ4s79dYeehFj3wCV2zdTf3L0u-wwQ-sdKA@mail.gmail.com>
Message-ID: <A212B88E-95A3-4C04-81A4-30233C3E345D@mcs.anl.gov>


  Also always use -snes_linesearch_monitor in these cases.

  Also when something goes wrong in a nonlinear solver with a fancy preconditioner on the linear solver it is good to make a run with a direct solver to make sure the issue is not with the preconditioner; so -pc_type lu


> On May 15, 2015, at 2:04 PM, Dmitry Karpeyev <karpeev at mcs.anl.gov> wrote:
> 
> What does -ksp_monitor_true_residual show?
> 
> On Fri, May 15, 2015 at 1:45 PM Fande Kong <fande.kong at colorado.edu> wrote:
> Hi all,
> 
> I was wondering why we will get a zero residual when solving the linear step of a nonlinear problem? I am solving a fluid-structure interaction problems using inexact Newton method, within which fGMRES is used together with a multilevel Schwarz preconditioner. The algorithm works well when the mesh is small, but it catched a zero residual issue when we turn to use a little bit large mesh.  The output is listed as follows:
> 
> 0 SNES Function norm 1.291775443677e-02 
>   Jacobian: rows 5575441  Column 5575441
> call DMGetInterpolation_SpmcsDMMesh 
>   Jacobian: rows 1298448  Column 1298448
>     0 KSP Residual norm 1.291775443677e-02 
>     1 KSP Residual norm 0.000000000000e+00 
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0
> 
> 
> Any suggestions? Thanks.
> 
> Fande,
> 
> 
> 


From venkateshgk.j at gmail.com  Fri May 15 14:46:03 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 16 May 2015 01:16:03 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hP6uhODEgD3PH=FpF8pCxkXNTCokEZnxvFN+MoVU18_PQ@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
	<618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
	<CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>
	<ADA696F0-6CD3-48EC-89C5-3BE5BDD06F70@dsic.upv.es>
	<CABzT7hP6uhODEgD3PH=FpF8pCxkXNTCokEZnxvFN+MoVU18_PQ@mail.gmail.com>
Message-ID: <CABzT7hPJGjZFQyLdmkns2igMpXQE+XKcg6oYnDPnc_qqsECK4A@mail.gmail.com>

Hi, I am now running the solver with the mumps

/ex7 -f1 adt -f2 bdt -st_type sinvert -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_package mumps -evecs outvec

However, my eigenvalue is not right by 0.5% and the eigenvector is not
correct when I plotted it.
It has to match the matlab result as it is a benchmark from a book.

Kindly advice.

Venkatesh



On Fri, May 15, 2015 at 8:24 PM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Ah. I made mistake by looking at the old manual. I am now reading the
> latest version of the manual.
>
> I will do this and get back.
>
> Venkatesh
>
> On Fri, May 15, 2015 at 6:14 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>
>> You have to choose MUMPS as the solver. ONCE AGAIN, read section 3.4.1 of
>> the users manual.
>> Jose
>>
>>
>> El 15/05/2015, a las 13:48, venkatesh g escribi?:
>>
>> > Ok, I followed this process and compiled ex7 successfully.
>> >
>> > Now I give the command: './ex7 -f1 A11 -f2 B11 -eps_nev 4 -eps_target
>> 0.5 -st_type sinvert'
>> >
>> > The error is the zero pivot problem:
>> >
>> >  Reading COMPLEX matrices from binary files...
>> > [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> > [0]PETSC ERROR: Detected zero pivot in LU factorization:
>> > see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
>> > [0]PETSC ERROR: Zero pivot row 420 value 0 tolerance 2.22045e-14!
>> > [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
>> 22:15:24 CDT 2013
>> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> > [0]PETSC ERROR: See docs/index.html for manual pages.
>> > [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
>> venkatesh Fri May 15 17:07:02 2015
>> > [0]PETSC ERROR: Libraries linked from
>> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
>> > [0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
>> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
>> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
>> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
>> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
>> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
>> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
>> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
>> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
>> --with-scalar-type=complex --with-fortran-kernels=generic
>> --with-large-file-io=1
>> > [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [0]PETSC ERROR: MatPivotCheck_none() line 583 in
>> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
>> > [0]PETSC ERROR: MatPivotCheck() line 602 in
>> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
>> > [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 570 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/seq/aijfact.c
>> > [0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
>> > [0]PETSC ERROR: PCSetUp_LU() line 160 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
>> > [0]PETSC ERROR: PCSetUp() line 832 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
>> > [0]PETSC ERROR: KSPSetUp() line 278 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
>> > [0]PETSC ERROR: PCSetUp_Redundant() line 176 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/redundant/redundant.c
>> > [0]PETSC ERROR: PCSetUp() line 832 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
>> > [0]PETSC ERROR: KSPSetUp() line 278 in
>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
>> > [0]PETSC ERROR: STSetUp_Sinvert() line 153 in
>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
>> > [0]PETSC ERROR: STSetUp() line 280 in
>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
>> > [0]PETSC ERROR: EPSSetUp() line 204 in
>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
>> > [0]PETSC ERROR: EPSSolve() line 109 in
>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
>> > [0]PETSC ERROR: main() line 148 in ex7.c
>> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
>> > [unset]: aborting job:
>> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
>> >
>> > Venkatesh
>> >
>> > On Fri, May 15, 2015 at 4:22 PM, Jose E. Roman <jroman at dsic.upv.es>
>> wrote:
>> >
>> > El 15/05/2015, a las 12:49, venkatesh g escribi?:
>> >
>> > > Hi,
>> > > 'make test' was successful on the $PETSC_DIR.
>> > > I configured MUMPS, metis, parmetis, scalapack through
>> '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..
>> > >
>> > > I will anyway download the new version and start installing it now.,
>> Is it due to some flag issue ? I have attached my makefile.
>> > >
>> > > Venkatesh
>> > >
>> >
>> > You must link with all libraries, otherwise you will get undefined
>> symbols. The easiest way is to include PETSc and SLEPc makefile
>> definitions. See for instance section 1.4.2 of SLEPc's users manual.
>> >
>> > Jose
>> >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/8837356b/attachment-0001.html>

From Sanjay.Kharche at manchester.ac.uk  Fri May 15 16:43:57 2015
From: Sanjay.Kharche at manchester.ac.uk (Sanjay Kharche)
Date: Fri, 15 May 2015 21:43:57 +0000
Subject: [petsc-users] convection terms in heat equation
Message-ID: <E8F644D1675C06428BC49CCCDF77EEEC453177A4@MBXP03.ds.man.ac.uk>


Dear All

When I put in my convection terms, my solution is not well behaved. Following is an elaboration. I am wondering if I missed something in Petsc.

I am solving the 3D heat equation with convection terms using second order finite differences and the boundary conditions are 1st order. It is solved on an uneven geometry defined inside a box, where a voltage variable diffuses. I am using a ts solver. The 3D DMDA vector uses a star stencil (I have also tried the box stencil). I assign the RHS at FD nodes where there is geometry. I get a stable solution if I do not have convection terms. However, when I use the convection terms, an error seems to build up. Eventually, the non-geometry part of the box also starts having a non-zero voltage. I checked my FD scheme for errors in terms. I also tested it out on a non-petsc explicit solver. The scheme is correctly written in the RHS function. The explicit solver is stable for the integration parameters I am using. Can you suggest what could be missing in terms of Petsc or otherwise?

cheers
Sanjay

From knepley at gmail.com  Fri May 15 17:00:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 15 May 2015 17:00:04 -0500
Subject: [petsc-users] convection terms in heat equation
In-Reply-To: <E8F644D1675C06428BC49CCCDF77EEEC453177A4@MBXP03.ds.man.ac.uk>
References: <E8F644D1675C06428BC49CCCDF77EEEC453177A4@MBXP03.ds.man.ac.uk>
Message-ID: <CAMYG4Gmd+kdenLok5x_S41wZxMcLW1a+GwgqH+a=xE6YELv5vQ@mail.gmail.com>

On Fri, May 15, 2015 at 4:43 PM, Sanjay Kharche <
Sanjay.Kharche at manchester.ac.uk> wrote:

>
> Dear All
>
> When I put in my convection terms, my solution is not well behaved.
> Following is an elaboration. I am wondering if I missed something in Petsc.
>
> I am solving the 3D heat equation with convection terms using second order
> finite differences and the boundary conditions are 1st order. It is solved
> on an uneven geometry defined inside a box, where a voltage variable
> diffuses. I am using a ts solver. The 3D DMDA vector uses a star stencil (I
> have also tried the box stencil). I assign the RHS at FD nodes where there
> is geometry. I get a stable solution if I do not have convection terms.
> However, when I use the convection terms, an error seems to build up.
> Eventually, the non-geometry part of the box also starts having a non-zero
> voltage. I checked my FD scheme for errors in terms. I also tested it out
> on a non-petsc explicit solver. The scheme is correctly written in the RHS
> function. The explicit solver is stable for the integration parameters I am
> using. Can you suggest what could be missing in terms of Petsc or otherwise?
>

Use the Method of Manufactured Solutions to check your residual function.

  Matt


> cheers
> Sanjay
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150515/b5c7cd43/attachment.html>

From mlohry at princeton.edu  Fri May 15 18:37:43 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Fri, 15 May 2015 19:37:43 -0400
Subject: [petsc-users] how to set the newton trust region size
Message-ID: <55568347.5040402@princeton.edu>

I'm having an issue with SNES where the regular newton line search 
severely overshoots the solution update, ending up with irrecoverable 
values (negative densities and temperatures in a fluids code), so it 
seems newtonTR is a better way to go.

the manual says to set delta0 using -snes_tr_delta0, where the trust 
region radius is delta = delta0 * norm2( F ). My F values aren't well 
scaled and have pretty different values so norm2 of the whole thing 
isn't particularly meaningful. Ideally I'd like to tell SNES to limit 
the update to say, 10% of the current value of any given unknown. Is 
this possible?

From venkateshgk.j at gmail.com  Sat May 16 01:02:52 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 16 May 2015 11:32:52 +0530
Subject: [petsc-users] SLEPc Eigenvalue problem - Zero Pivot
In-Reply-To: <CABzT7hPJGjZFQyLdmkns2igMpXQE+XKcg6oYnDPnc_qqsECK4A@mail.gmail.com>
References: <CABzT7hNTjHPNSDtCxwpWBO4SFL6a0UFd+4q31nHa49w2R9uu2A@mail.gmail.com>
	<5F9A09C1-075C-4306-9447-B0F754D610B7@dsic.upv.es>
	<CABzT7hMFoRgCBBLCNcM7V2uhM--2myF6=kzF-pVZDafyXnx2vQ@mail.gmail.com>
	<8E0C34B0-FD35-4578-AB84-03B7B99E91DE@dsic.upv.es>
	<CABzT7hO4pKyLsXRzysJD49nXTGX9pSChPvT+NVYCM7nk96HovQ@mail.gmail.com>
	<50671AB9-E91E-4D9E-AA33-5262CECD7A86@dsic.upv.es>
	<CABzT7hMVvZ1-_f_MzA7tsgS+NbXpsksV-KKvgENqdaF9ux0wBw@mail.gmail.com>
	<B1EC7C86-5FC3-4847-A4E6-135C44CF7555@dsic.upv.es>
	<CABzT7hNeF7atSexOk0bbVKpu2yPTpm1i15gCw2Yk3ef_uKhKMw@mail.gmail.com>
	<4DAF9C2F-2F4D-47E8-81C4-1F766C81AF8C@dsic.upv.es>
	<CABzT7hM0Uq19D+XXvST1iBYafZwJZZ=dqMU1CQRa+e1TF9w8Ow@mail.gmail.com>
	<9724AEF5-A868-42CE-8F62-E440F90DA117@dsic.upv.es>
	<CABzT7hPSZ0mS23yHBg36n6f6N4NtzoHKpJocHz7T+HA5bJzrBg@mail.gmail.com>
	<618DF142-6ABD-4559-AE14-33D114BC410B@dsic.upv.es>
	<CABzT7hPa9xhuhimMAE+6og=4MGn7qzEnSE4L-UJue1T1EOWCDA@mail.gmail.com>
	<ADA696F0-6CD3-48EC-89C5-3BE5BDD06F70@dsic.upv.es>
	<CABzT7hP6uhODEgD3PH=FpF8pCxkXNTCokEZnxvFN+MoVU18_PQ@mail.gmail.com>
	<CABzT7hPJGjZFQyLdmkns2igMpXQE+XKcg6oYnDPnc_qqsECK4A@mail.gmail.com>
Message-ID: <CABzT7hNgQLgY56JqSK_69adwQjBOf5Fw-jDT=c7jYPwk=7rxOw@mail.gmail.com>

Hi, I managed to plot it in matlab with the PetscBinaryRead with options:
vc=PetscBinaryRead('outvec','complex',true);

I missed the option 'complex' previously.

It is working fine now. I will ask another separate query so that this can
be closed.

Venkatesh

On Sat, May 16, 2015 at 1:16 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi, I am now running the solver with the mumps
>
> /ex7 -f1 adt -f2 bdt -st_type sinvert -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps -evecs outvec
>
> However, my eigenvalue is not right by 0.5% and the eigenvector is not
> correct when I plotted it.
> It has to match the matlab result as it is a benchmark from a book.
>
> Kindly advice.
>
> Venkatesh
>
>
>
> On Fri, May 15, 2015 at 8:24 PM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Ah. I made mistake by looking at the old manual. I am now reading the
>> latest version of the manual.
>>
>> I will do this and get back.
>>
>> Venkatesh
>>
>> On Fri, May 15, 2015 at 6:14 PM, Jose E. Roman <jroman at dsic.upv.es>
>> wrote:
>>
>>> You have to choose MUMPS as the solver. ONCE AGAIN, read section 3.4.1
>>> of the users manual.
>>> Jose
>>>
>>>
>>> El 15/05/2015, a las 13:48, venkatesh g escribi?:
>>>
>>> > Ok, I followed this process and compiled ex7 successfully.
>>> >
>>> > Now I give the command: './ex7 -f1 A11 -f2 B11 -eps_nev 4 -eps_target
>>> 0.5 -st_type sinvert'
>>> >
>>> > The error is the zero pivot problem:
>>> >
>>> >  Reading COMPLEX matrices from binary files...
>>> > [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> > [0]PETSC ERROR: Detected zero pivot in LU factorization:
>>> > see http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot!
>>> > [0]PETSC ERROR: Zero pivot row 420 value 0 tolerance 2.22045e-14!
>>> > [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11
>>> 22:15:24 CDT 2013
>>> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> > [0]PETSC ERROR: See docs/index.html for manual pages.
>>> > [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> > [0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in
>>> by venkatesh Fri May 15 17:07:02 2015
>>> > [0]PETSC ERROR: Libraries linked from
>>> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
>>> > [0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
>>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
>>> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
>>> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
>>> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
>>> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
>>> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
>>> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
>>> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
>>> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
>>> --with-scalar-type=complex --with-fortran-kernels=generic
>>> --with-large-file-io=1
>>> > [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> > [0]PETSC ERROR: MatPivotCheck_none() line 583 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
>>> > [0]PETSC ERROR: MatPivotCheck() line 602 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/include/petsc-private/matimpl.h
>>> > [0]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 570 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/seq/aijfact.c
>>> > [0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
>>> > [0]PETSC ERROR: PCSetUp_LU() line 160 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
>>> > [0]PETSC ERROR: PCSetUp() line 832 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
>>> > [0]PETSC ERROR: KSPSetUp() line 278 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
>>> > [0]PETSC ERROR: PCSetUp_Redundant() line 176 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/redundant/redundant.c
>>> > [0]PETSC ERROR: PCSetUp() line 832 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
>>> > [0]PETSC ERROR: KSPSetUp() line 278 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
>>> > [0]PETSC ERROR: STSetUp_Sinvert() line 153 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
>>> > [0]PETSC ERROR: STSetUp() line 280 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
>>> > [0]PETSC ERROR: EPSSetUp() line 204 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
>>> > [0]PETSC ERROR: EPSSolve() line 109 in
>>> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
>>> > [0]PETSC ERROR: main() line 148 in ex7.c
>>> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
>>> > [unset]: aborting job:
>>> > application called MPI_Abort(MPI_COMM_WORLD, 71) - process 0
>>> >
>>> > Venkatesh
>>> >
>>> > On Fri, May 15, 2015 at 4:22 PM, Jose E. Roman <jroman at dsic.upv.es>
>>> wrote:
>>> >
>>> > El 15/05/2015, a las 12:49, venkatesh g escribi?:
>>> >
>>> > > Hi,
>>> > > 'make test' was successful on the $PETSC_DIR.
>>> > > I configured MUMPS, metis, parmetis, scalapack through
>>> '--download-mumps=/path/MUMPS_4.10.0-p3.tar.gz' and etc..
>>> > >
>>> > > I will anyway download the new version and start installing it now.,
>>> Is it due to some flag issue ? I have attached my makefile.
>>> > >
>>> > > Venkatesh
>>> > >
>>> >
>>> > You must link with all libraries, otherwise you will get undefined
>>> symbols. The easiest way is to include PETSc and SLEPc makefile
>>> definitions. See for instance section 1.4.2 of SLEPc's users manual.
>>> >
>>> > Jose
>>> >
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/b4e8f140/attachment.html>

From venkateshgk.j at gmail.com  Sat May 16 03:49:55 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 16 May 2015 14:19:55 +0530
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
Message-ID: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>

Hi,

I am trying to solving AX=lambda BX

A and B are of size 57000 x 57000

Currently I solve my small matrices using -st_type sinvert and -eps_nev 1

So I get 1 eigenvalue and eigenvector.

So for the 57000x57000 large matrix problem, the unknown eigenvector is
57000x1

Should I configure the petsc as '--with-64-bit-indices=1' for this problem
?
Is the unknowns being considered as the entire eigenspace 57000x57000 ?

kindly let me know.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/5281fd96/attachment-0001.html>

From venkateshgk.j at gmail.com  Sat May 16 07:08:55 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 16 May 2015 17:38:55 +0530
Subject: [petsc-users] MUMPS error
Message-ID: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>

Hi,
I am trying to solving AX=lambda BX eigenvalue problem.

A and B are of sizes 3600x3600

I run with this command :

'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert -eps_max_it
5000 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package
mumps'

I get this error: (I get result only when I give 1 or 2 processors)
Reading COMPLEX matrices from binary files...
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFO(1)=-9, INFO(2)=2024
!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
CDT 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
venkatesh Sat May 16 17:20:54 2015
[0]PETSC ERROR: Libraries linked from
/cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
[0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Error in external library!
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFO(1)=-1, INFO(2)=0
!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
CDT 2013
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
--with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
--download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
--download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
--download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
--download-blacs=/cluster/share/apps/blacs-dev.tar.gz
--download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
--download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
--download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
--with-scalar-type=complex --with-fortran-kernels=generic
--with-large-file-io=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
venkatesh Sat May 16 17:20:54 2015
[2]PETSC ERROR: Libraries linked from
/cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
[2]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
[2]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
--with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
--download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
--download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
--download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
--download-blacs=/cluster/share/apps/blacs-dev.tar.gz
--download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
--download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
--download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
--with-scalar-type=complex --with-fortran-kernels=generic
--with-large-file-io=1
[0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCSetUp_LU() line 160 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: [2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
[2]PETSC ERROR: PCSetUp() line 832 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 278 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
MatLUFactorNumeric() line 2803 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
[2]PETSC ERROR: PCSetUp_LU() line 160 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
[2]PETSC ERROR: [0]PETSC ERROR: STSetUp_Sinvert() line 153 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: PCSetUp() line 832 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: KSPSetUp() line 278 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: STSetUp_Sinvert() line 153 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
[2]PETSC ERROR: STSetUp() line 280 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
[0]PETSC ERROR: EPSSetUp() line 204 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
[0]PETSC ERROR: EPSSolve() line 109 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
STSetUp() line 280 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
[2]PETSC ERROR: EPSSetUp() line 204 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
[2]PETSC ERROR: EPSSolve() line 109 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
[0]PETSC ERROR: main() line 148 in ex7.c
[2]PETSC ERROR: main() line 148 in ex7.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
[cli_2]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
[6]PETSC ERROR: --------------------- Error Message
------------------------------------
[6]PETSC ERROR: Error in external library!
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
INFO(1)=-1, INFO(2)=0
!
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
CDT 2013
[6]PETSC ERROR: See docs/changes/index.html for recent updates.
[6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[6]PETSC ERROR: See docs/index.html for manual pages.
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
venkatesh Sat May 16 17:20:54 2015
[6]PETSC ERROR: Libraries linked from
/cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
[6]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
[6]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
--with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
--download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
--download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
--download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
--download-blacs=/cluster/share/apps/blacs-dev.tar.gz
--download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
--download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
--download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
--with-scalar-type=complex --with-fortran-kernels=generic
--with-large-file-io=1
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: MatLUFactorNumeric() line 2803 in
/cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
[6]PETSC ERROR: PCSetUp_LU() line 160 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: PCSetUp() line 832 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: KSPSetUp() line 278 in
/cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: STSetUp_Sinvert() line 153 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
[6]PETSC ERROR: STSetUp() line 280 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
[6]PETSC ERROR: EPSSetUp() line 204 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
[6]PETSC ERROR: EPSSolve() line 109 in
/cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
[6]PETSC ERROR: main() line 148 in ex7.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
[cli_6]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6

Kindly help.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/022070bd/attachment.html>

From knepley at gmail.com  Sat May 16 09:38:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 16 May 2015 09:38:56 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
Message-ID: <CAMYG4GmdnUKRfKD4du3rP2A7qUXrbfkzAhrNjMsNnWCmLLTo0w@mail.gmail.com>

On Sat, May 16, 2015 at 7:08 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
> I am trying to solving AX=lambda BX eigenvalue problem.
>
> A and B are of sizes 3600x3600
>
> I run with this command :
>
> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert -eps_max_it
> 5000 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package
> mumps'
>
> I get this error: (I get result only when I give 1 or 2 processors)
> Reading COMPLEX matrices from binary files...
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFO(1)=-9, INFO(2)=2024
> !
>

>From the MUMPS manual, this appears to be insufficient workspace. In the
manual, they describe inputs that
can increase the workspace. You should be able to give those on the command
line (see MUMPS documentation
in PETSc).

  Thanks,

    Matt


> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
> CDT 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
> venkatesh Sat May 16 17:20:54 2015
> [0]PETSC ERROR: Libraries linked from
> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
> [0]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
> [2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Error in external library!
> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFO(1)=-1, INFO(2)=0
> !
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
> CDT 2013
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
> --with-scalar-type=complex --with-fortran-kernels=generic
> --with-large-file-io=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
> venkatesh Sat May 16 17:20:54 2015
> [2]PETSC ERROR: Libraries linked from
> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
> [2]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
> [2]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
> --with-scalar-type=complex --with-fortran-kernels=generic
> --with-large-file-io=1
> [0]PETSC ERROR: MatLUFactorNumeric() line 2803 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
> [0]PETSC ERROR: PCSetUp_LU() line 160 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
> [0]PETSC ERROR: [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
> [2]PETSC ERROR: PCSetUp() line 832 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 278 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> MatLUFactorNumeric() line 2803 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
> [2]PETSC ERROR: PCSetUp_LU() line 160 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
> [2]PETSC ERROR: [0]PETSC ERROR: STSetUp_Sinvert() line 153 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
> [0]PETSC ERROR: PCSetUp() line 832 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> [2]PETSC ERROR: KSPSetUp() line 278 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> [2]PETSC ERROR: STSetUp_Sinvert() line 153 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
> [2]PETSC ERROR: STSetUp() line 280 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
> [0]PETSC ERROR: EPSSetUp() line 204 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
> [0]PETSC ERROR: EPSSolve() line 109 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
> STSetUp() line 280 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
> [2]PETSC ERROR: EPSSetUp() line 204 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
> [2]PETSC ERROR: EPSSolve() line 109 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
> [0]PETSC ERROR: main() line 148 in ex7.c
> [2]PETSC ERROR: main() line 148 in ex7.c
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
> [cli_2]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
> [cli_0]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
> [6]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [6]PETSC ERROR: Error in external library!
> [6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFO(1)=-1, INFO(2)=0
> !
> [6]PETSC ERROR:
> ------------------------------------------------------------------------
> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24
> CDT 2013
> [6]PETSC ERROR: See docs/changes/index.html for recent updates.
> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [6]PETSC ERROR: See docs/index.html for manual pages.
> [6]PETSC ERROR:
> ------------------------------------------------------------------------
> [6]PETSC ERROR: ./ex7 on a linux-gnu named earth.ceas.iisc.ernet.in by
> venkatesh Sat May 16 17:20:54 2015
> [6]PETSC ERROR: Libraries linked from
> /cluster/share/venkatesh/petsc-3.3-p7/linux-gnu-intel-cmplx-batch/lib
> [6]PETSC ERROR: Configure run at Fri May 15 15:10:57 2015
> [6]PETSC ERROR: Configure options --with-cc=gcc --with-fc=ifort
> --with-blas-lapack-dir=/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
> --download-mpich=/cluster/share/apps/mpich2-1.4.1p1.tar.gz
> --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz
> --download-scalapack=/cluster/share/apps/SCALAPACK-1.7.tar.gz
> --download-blacs=/cluster/share/apps/blacs-dev.tar.gz
> --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p3.tar.gz
> --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz
> --download-cmake=/cluster/share/apps/cmake-2.8.7.tar.gz
> --with-scalar-type=complex --with-fortran-kernels=generic
> --with-large-file-io=1
> [6]PETSC ERROR:
> ------------------------------------------------------------------------
> [6]PETSC ERROR: MatFactorNumeric_MUMPS() line 670 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/impls/aij/mpi/mumps/mumps.c
> [6]PETSC ERROR: MatLUFactorNumeric() line 2803 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/mat/interface/matrix.c
> [6]PETSC ERROR: PCSetUp_LU() line 160 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/impls/factor/lu/lu.c
> [6]PETSC ERROR: PCSetUp() line 832 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/pc/interface/precon.c
> [6]PETSC ERROR: KSPSetUp() line 278 in
> /cluster/share/venkatesh/petsc-3.3-p7/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: STSetUp_Sinvert() line 153 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/impls/sinvert/sinvert.c
> [6]PETSC ERROR: STSetUp() line 280 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/st/interface/stsolve.c
> [6]PETSC ERROR: EPSSetUp() line 204 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/setup.c
> [6]PETSC ERROR: EPSSolve() line 109 in
> /cluster/share/venkatesh/petsc-3.3-p7/slepc-3.3-p4/src/eps/interface/solve.c
> [6]PETSC ERROR: main() line 148 in ex7.c
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
> [cli_6]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
>
> Kindly help.
>
> Venkatesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/4d8d12ad/attachment-0001.html>

From knepley at gmail.com  Sat May 16 09:33:41 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 16 May 2015 09:33:41 -0500
Subject: [petsc-users] how to set the newton trust region size
In-Reply-To: <55568347.5040402@princeton.edu>
References: <55568347.5040402@princeton.edu>
Message-ID: <CAMYG4G=MYsZdGsU-u+aQHiNU6ZQP6KeXAgB6GpYueqkdi5u5GQ@mail.gmail.com>

On Fri, May 15, 2015 at 6:37 PM, Mark Lohry <mlohry at princeton.edu> wrote:

> I'm having an issue with SNES where the regular newton line search
> severely overshoots the solution update, ending up with irrecoverable
> values (negative densities and temperatures in a fluids code), so it seems
> newtonTR is a better way to go.
>
> the manual says to set delta0 using -snes_tr_delta0, where the trust
> region radius is delta = delta0 * norm2( F ). My F values aren't well
> scaled and have pretty different values so norm2 of the whole thing isn't
> particularly meaningful. Ideally I'd like to tell SNES to limit the update
> to say, 10% of the current value of any given unknown. Is this possible?
>

1) The best thing you can do for solver convergence is to start with a
well-scaled system. Is there a reason why
you cannot non-dimensionalize your system?

2) Writing a custom line search is easier than messing with TR in my
experience

3) If you have constraints on the variables, you might benefit from using
SNESVI

  Thanks,

      Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/765c84c7/attachment.html>

From knepley at gmail.com  Sat May 16 09:35:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 16 May 2015 09:35:47 -0500
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
In-Reply-To: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
References: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
Message-ID: <CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>

On Sat, May 16, 2015 at 3:49 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
>
> I am trying to solving AX=lambda BX
>
> A and B are of size 57000 x 57000
>
> Currently I solve my small matrices using -st_type sinvert and -eps_nev 1
>
> So I get 1 eigenvalue and eigenvector.
>
> So for the 57000x57000 large matrix problem, the unknown eigenvector is
> 57000x1
>
> Should I configure the petsc as '--with-64-bit-indices=1' for this problem
> ?
>

It is not necessary.

   Matt


> Is the unknowns being considered as the entire eigenspace 57000x57000 ?
>
> kindly let me know.
>
> Venkatesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/951261dd/attachment.html>

From david.knezevic at akselos.com  Sat May 16 08:40:19 2015
From: david.knezevic at akselos.com (David Knezevic)
Date: Sat, 16 May 2015 09:40:19 -0400
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
Message-ID: <CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>

On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
> I am trying to solving AX=lambda BX eigenvalue problem.
>
> A and B are of sizes 3600x3600
>
> I run with this command :
>
> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert -eps_max_it
> 5000 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package
> mumps'
>
> I get this error: (I get result only when I give 1 or 2 processors)
> Reading COMPLEX matrices from binary files...
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> INFO(1)=-9, INFO(2)=2024
>


The MUMPS error types are described in Chapter 7 of the MUMPS manual. In
this case you have INFO(1)=-9, which is explained in the manual as:

"?9 Main internal real/complex workarray S too small. If INFO(2) is
positive, then the number of entries that are missing in S at the moment
when the error is raised is available in INFO(2). If INFO(2) is negative,
then its absolute value should be multiplied by 1 million. If an error ?9
occurs, the user should increase the value of ICNTL(14) before calling the
factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
ICNTL(23) should be increased."

This says that you should use ICTNL(14) to increase the working space size:

"ICNTL(14) is accessed by the host both during the analysis and the
factorization phases. It corresponds to the percentage increase in the
estimated working space. When significant extra fill-in is caused by
numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
the default value is 20 (which corresponds to a 20 % increase)."

So, for example, you can avoid this error via the following command line
argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we
allow a 30% increase in the workspace instead of the default 20%.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/e57c46bf/attachment.html>

From mlohry at princeton.edu  Sat May 16 11:15:53 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Sat, 16 May 2015 12:15:53 -0400
Subject: [petsc-users] how to set the newton trust region size
In-Reply-To: <CAMYG4G=MYsZdGsU-u+aQHiNU6ZQP6KeXAgB6GpYueqkdi5u5GQ@mail.gmail.com>
References: <55568347.5040402@princeton.edu>
	<CAMYG4G=MYsZdGsU-u+aQHiNU6ZQP6KeXAgB6GpYueqkdi5u5GQ@mail.gmail.com>
Message-ID: <55576D39.5080003@princeton.edu>

On 5/16/2015 10:33 AM, Matthew Knepley wrote:
>
> 1) The best thing you can do for solver convergence is to start with a 
> well-scaled system. Is there a reason why
> you cannot non-dimensionalize your system?
>

Laziness/time, but yes I'll do this. Though this is a viscous flow 
solver, so any non-dimensionalization will still lead to values 
differing by 5+ orders of magnitude in certain areas.


>
> 3) If you have constraints on the variables, you might benefit from 
> using SNESVI
>
>


Perfect, thanks I wasn't aware of this. So SNESVIComputeVariableBounds 
can take a function that say, prevents negative densities/pressures?

> 2) Writing a custom line search is easier than messing with TR in my 
> experience
>

I'm interested to hear more about this. I haven't done much work with 
newton methods on this type of problem where overshoots lead to fatal 
errors.

From knepley at gmail.com  Sat May 16 11:25:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 16 May 2015 11:25:58 -0500
Subject: [petsc-users] how to set the newton trust region size
In-Reply-To: <55576D39.5080003@princeton.edu>
References: <55568347.5040402@princeton.edu>
	<CAMYG4G=MYsZdGsU-u+aQHiNU6ZQP6KeXAgB6GpYueqkdi5u5GQ@mail.gmail.com>
	<55576D39.5080003@princeton.edu>
Message-ID: <CAMYG4GnVbRAR+KjjQLFXM5ZQOihntW85ZhoR3Jk-qu8uUApHYg@mail.gmail.com>

On Sat, May 16, 2015 at 11:15 AM, Mark Lohry <mlohry at princeton.edu> wrote:

> On 5/16/2015 10:33 AM, Matthew Knepley wrote:
>
>>
>> 1) The best thing you can do for solver convergence is to start with a
>> well-scaled system. Is there a reason why
>> you cannot non-dimensionalize your system?
>>
>>
> Laziness/time, but yes I'll do this. Though this is a viscous flow solver,
> so any non-dimensionalization will still lead to values differing by 5+
> orders of magnitude in certain areas.
>
>
>
>> 3) If you have constraints on the variables, you might benefit from using
>> SNESVI
>>
>>
>>
>
> Perfect, thanks I wasn't aware of this. So SNESVIComputeVariableBounds can
> take a function that say, prevents negative densities/pressures?
>

It only really handles primary variable bounds, but if you are solving for
densities and pressures, yes.


>  2) Writing a custom line search is easier than messing with TR in my
>> experience
>>
>>
> I'm interested to hear more about this. I haven't done much work with
> newton methods on this type of problem where overshoots lead to fatal
> errors.
>

You can imagine the very simplest thing you could do for SNESVI is take
each step produced by Newton
and scale it back until it lies in your "physical" region. This however can
degrade the second order
convergence of the method. SNESVI tries to be smarter about things to
maintain the convergence rate.

  Thanks,

    Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150516/b6c09bfc/attachment.html>

From venkateshgk.j at gmail.com  Sun May 17 01:38:51 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sun, 17 May 2015 12:08:51 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
Message-ID: <CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>

Hi, Thanks for the information. I now increased the workspace by adding
'-mat_mumps_icntl_14 100'

It works. However, the problem is, if I submit in 1 core I get the answer
in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
3500secs.

My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type
sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'

Kindly let me know.

Venkatesh



On Sat, May 16, 2015 at 7:10 PM, David Knezevic <david.knezevic at akselos.com>
wrote:

> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi,
>> I am trying to solving AX=lambda BX eigenvalue problem.
>>
>> A and B are of sizes 3600x3600
>>
>> I run with this command :
>>
>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package mumps'
>>
>> I get this error: (I get result only when I give 1 or 2 processors)
>> Reading COMPLEX matrices from binary files...
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Error in external library!
>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
>> INFO(1)=-9, INFO(2)=2024
>>
>
>
> The MUMPS error types are described in Chapter 7 of the MUMPS manual. In
> this case you have INFO(1)=-9, which is explained in the manual as:
>
> "?9 Main internal real/complex workarray S too small. If INFO(2) is
> positive, then the number of entries that are missing in S at the moment
> when the error is raised is available in INFO(2). If INFO(2) is negative,
> then its absolute value should be multiplied by 1 million. If an error ?9
> occurs, the user should increase the value of ICNTL(14) before calling the
> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
> ICNTL(23) should be increased."
>
> This says that you should use ICTNL(14) to increase the working space size:
>
> "ICNTL(14) is accessed by the host both during the analysis and the
> factorization phases. It corresponds to the percentage increase in the
> estimated working space. When significant extra fill-in is caused by
> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
> the default value is 20 (which corresponds to a 20 % increase)."
>
> So, for example, you can avoid this error via the following command line
> argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we
> allow a 30% increase in the workspace instead of the default 20%.
>
> David
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150517/06da7dad/attachment.html>

From venkateshgk.j at gmail.com  Sun May 17 02:02:16 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sun, 17 May 2015 12:32:16 +0530
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
In-Reply-To: <CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>
References: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
	<CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>
Message-ID: <CABzT7hN3yaoCYyNK46QBzdKeBHNyGR6PneWEqUJUuwmBJ2SjBg@mail.gmail.com>

Thank you. So the single eigenvector's size which is Nx1 is the number of
unknowns and should be less than 2^31-1 so as to avoid configuring with
'--with-64-bit-indices=1' ??


On Sat, May 16, 2015 at 8:05 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, May 16, 2015 at 3:49 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to solving AX=lambda BX
>>
>> A and B are of size 57000 x 57000
>>
>> Currently I solve my small matrices using -st_type sinvert and -eps_nev 1
>>
>> So I get 1 eigenvalue and eigenvector.
>>
>> So for the 57000x57000 large matrix problem, the unknown eigenvector is
>> 57000x1
>>
>> Should I configure the petsc as '--with-64-bit-indices=1' for this
>> problem ?
>>
>
> It is not necessary.
>
>    Matt
>
>
>> Is the unknowns being considered as the entire eigenspace 57000x57000 ?
>>
>> kindly let me know.
>>
>> Venkatesh
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150517/387f9c22/attachment.html>

From jroman at dsic.upv.es  Sun May 17 04:16:08 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Sun, 17 May 2015 11:16:08 +0200
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
In-Reply-To: <CABzT7hN3yaoCYyNK46QBzdKeBHNyGR6PneWEqUJUuwmBJ2SjBg@mail.gmail.com>
References: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
	<CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>
	<CABzT7hN3yaoCYyNK46QBzdKeBHNyGR6PneWEqUJUuwmBJ2SjBg@mail.gmail.com>
Message-ID: <A4C36052-A95A-4804-A4C9-DF05E1481CAE@dsic.upv.es>

http://www.mcs.anl.gov/petsc/documentation/faq.html#with-64-bit-indices



El 17/05/2015, a las 09:02, venkatesh g escribi?:

> Thank you. So the single eigenvector's size which is Nx1 is the number of unknowns and should be less than 2^31-1 so as to avoid configuring with '--with-64-bit-indices=1' ??
> 
> 
> On Sat, May 16, 2015 at 8:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Sat, May 16, 2015 at 3:49 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> Hi,
> 
> I am trying to solving AX=lambda BX
> 
> A and B are of size 57000 x 57000
> 
> Currently I solve my small matrices using -st_type sinvert and -eps_nev 1
> 
> So I get 1 eigenvalue and eigenvector.
> 
> So for the 57000x57000 large matrix problem, the unknown eigenvector is 57000x1
> 
> Should I configure the petsc as '--with-64-bit-indices=1' for this problem ? 
> 
> It is not necessary.
> 
>    Matt
>  
> Is the unknowns being considered as the entire eigenspace 57000x57000 ?
> 
> kindly let me know.
> 
> Venkatesh
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From knepley at gmail.com  Sun May 17 07:43:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 17 May 2015 07:43:16 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
Message-ID: <CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>

On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi, Thanks for the information. I now increased the workspace by adding
> '-mat_mumps_icntl_14 100'
>
> It works. However, the problem is, if I submit in 1 core I get the answer
> in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
> 3500secs.
>

Send the output of -log_summary for all performance queries. Otherwise we
are just guessing.

    Matt

My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type
> sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>
> Kindly let me know.
>
> Venkatesh
>
>
>
> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
> david.knezevic at akselos.com> wrote:
>
>> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am trying to solving AX=lambda BX eigenvalue problem.
>>>
>>> A and B are of sizes 3600x3600
>>>
>>> I run with this command :
>>>
>>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>> -st_pc_factor_mat_solver_package mumps'
>>>
>>> I get this error: (I get result only when I give 1 or 2 processors)
>>> Reading COMPLEX matrices from binary files...
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Error in external library!
>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFO(1)=-9, INFO(2)=2024
>>>
>>
>>
>> The MUMPS error types are described in Chapter 7 of the MUMPS manual. In
>> this case you have INFO(1)=-9, which is explained in the manual as:
>>
>> "?9 Main internal real/complex workarray S too small. If INFO(2) is
>> positive, then the number of entries that are missing in S at the moment
>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>> then its absolute value should be multiplied by 1 million. If an error ?9
>> occurs, the user should increase the value of ICNTL(14) before calling the
>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>> ICNTL(23) should be increased."
>>
>> This says that you should use ICTNL(14) to increase the working space
>> size:
>>
>> "ICNTL(14) is accessed by the host both during the analysis and the
>> factorization phases. It corresponds to the percentage increase in the
>> estimated working space. When significant extra fill-in is caused by
>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>> the default value is 20 (which corresponds to a 20 % increase)."
>>
>> So, for example, you can avoid this error via the following command line
>> argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we
>> allow a 30% increase in the workspace instead of the default 20%.
>>
>> David
>>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150517/a21db20e/attachment.html>

From venkateshgk.j at gmail.com  Mon May 18 08:29:08 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Mon, 18 May 2015 18:59:08 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
Message-ID: <CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>

Hi I have attached the performance logs for 2 jobs on different processors.
I had to increase the workspace icntl(14) when I submit on more cores since
it is failing with small value of icntl(14).

1. performance_log1.txt is run on 8 cores (option given -mat_mumps_icntl_14
200)
2. performance_log2.txt is run on 2 cores (option given -mat_mumps_icntl_14
85  )

Venkatesh

On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi, Thanks for the information. I now increased the workspace by adding
>> '-mat_mumps_icntl_14 100'
>>
>> It works. However, the problem is, if I submit in 1 core I get the answer
>> in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>> 3500secs.
>>
>
> Send the output of -log_summary for all performance queries. Otherwise we
> are just guessing.
>
>     Matt
>
> My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type
>> sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>
>> Kindly let me know.
>>
>> Venkatesh
>>
>>
>>
>> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>>
>>> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I am trying to solving AX=lambda BX eigenvalue problem.
>>>>
>>>> A and B are of sizes 3600x3600
>>>>
>>>> I run with this command :
>>>>
>>>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>> -st_pc_factor_mat_solver_package mumps'
>>>>
>>>> I get this error: (I get result only when I give 1 or 2 processors)
>>>> Reading COMPLEX matrices from binary files...
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> [0]PETSC ERROR: Error in external library!
>>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>
>>>
>>>
>>> The MUMPS error types are described in Chapter 7 of the MUMPS manual. In
>>> this case you have INFO(1)=-9, which is explained in the manual as:
>>>
>>> "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>> positive, then the number of entries that are missing in S at the moment
>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>> ICNTL(23) should be increased."
>>>
>>> This says that you should use ICTNL(14) to increase the working space
>>> size:
>>>
>>> "ICNTL(14) is accessed by the host both during the analysis and the
>>> factorization phases. It corresponds to the percentage increase in the
>>> estimated working space. When significant extra fill-in is caused by
>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>
>>> So, for example, you can avoid this error via the following command line
>>> argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we
>>> allow a 30% increase in the workspace instead of the default 20%.
>>>
>>> David
>>>
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/d2917e37/attachment-0001.html>
-------------- next part --------------
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex7 on a linux-gnu named compute-0-0.local with 8 processors, by venkatesh Mon May 18 23:46:29 2015
Using Petsc Release Version 3.5.3, Jan, 31, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           6.437e+02      1.00000   6.437e+02
Objects:              6.200e+01      1.03333   6.100e+01
Flops:                2.592e+09      1.17536   2.284e+09  1.827e+10
Flops/sec:            4.027e+06      1.17536   3.548e+06  2.838e+07
MPI Messages:         6.246e+04      6.48624   2.163e+04  1.730e+05
MPI Message Lengths:  3.670e+08      5.23102   5.572e+03  9.643e+08
MPI Reductions:       2.732e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.4373e+02 100.0%  1.8271e+10 100.0%  1.730e+05 100.0%  5.572e+03      100.0%  2.731e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult             9604 1.0 1.8260e+0019.8 3.88e+08512.9 1.9e+04 6.4e+03 0.0e+00  0  3 11 13  0   0  3 11 13  0   346
MatSolve            9600 1.0 6.3221e+02 1.0 0.00e+00 0.0 1.5e+05 5.3e+03 9.6e+03 98  0 89 85 35  98  0 89 85 35     0
MatLUFactorSym         1 1.0 9.9645e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 4.3477e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatAssemblyBegin       2 1.0 8.7760e-02 8.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         2 1.0 2.3864e-02 1.7 0.00e+00 0.0 8.8e+01 5.8e+02 1.6e+01  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 3.0994e-06 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 5.9128e-05 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLoad                2 1.0 2.1068e-01 1.0 0.00e+00 0.0 1.3e+02 1.7e+05 2.6e+01  0  0  0  2  0   0  0  0  2  0     0
VecNorm                4 1.0 2.6133e-03 2.3 1.44e+04 1.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0    44
VecCopy             1202 1.0 2.3665e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              9605 1.0 2.6972e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                2 1.0 9.0599e-06 1.8 7.20e+03 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6358
VecScatterBegin    28804 1.0 5.1131e+0020.6 0.00e+00 0.0 1.7e+05 5.5e+03 9.6e+03  0  0100 98 35   0  0100 98 35     0
VecScatterEnd      19204 1.0 3.1801e+0019.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
EPSSetUp               1 1.0 5.3490e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+01  1  0  0  0  0   1  0  0  0  0     0
EPSSolve               1 1.0 6.4351e+02 1.0 2.59e+09 1.2 1.7e+05 5.5e+03 2.7e+04100100100 98100 100100100 98100    28
STSetUp                1 1.0 5.3477e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.3e+01  1  0  0  0  0   1  0  0  0  0     0
STApply             9600 1.0 6.3276e+02 1.0 3.84e+08 0.0 1.7e+05 5.5e+03 9.6e+03 98  3100 98 35  98  3100 98 35     1
STMatSolve          9600 1.0 6.3261e+02 1.0 0.00e+00 0.0 1.5e+05 5.3e+03 9.6e+03 98  0 89 85 35  98  0 89 85 35     0
BVCopy              1201 1.0 5.5201e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BVMult             18857 1.0 6.8241e-01 1.1 1.34e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0 59  0  0  0   0 59  0  0  0 15686
BVDot              17657 1.0 7.0668e+00 1.7 8.49e+08 1.0 0.0e+00 0.0e+00 1.8e+04  1 37  0  0 65   1 37  0  0 65   962
BVOrthogonalize     9601 1.0 7.5371e+00 1.6 1.64e+09 1.0 0.0e+00 0.0e+00 1.8e+04  1 72  0  0 65   1 72  0  0 65  1736
BVScale             9601 1.0 2.8411e-02 1.7 1.73e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4866
BVSetRandom            1 1.0 1.6403e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSSolve             1199 1.0 2.6696e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSVectors           1202 1.0 9.8028e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSOther             1199 1.0 1.0596e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            9600 1.0 6.3237e+02 1.0 0.00e+00 0.0 1.5e+05 5.3e+03 9.6e+03 98  0 89 85 35  98  0 89 85 35     0
PCSetUp                1 1.0 5.3474e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01  1  0  0  0  0   1  0  0  0  0     0
PCApply             9600 1.0 6.3223e+02 1.0 0.00e+00 0.0 1.5e+05 5.3e+03 9.6e+03 98  0 89 85 35  98  0 89 85 35     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     3              2         1504     0
              Matrix    11             11      1063160     0
              Vector    23             23       336320     0
      Vector Scatter     4              4         4352     0
           Index Set    11             11        27628     0
Eigenvalue Problem Solver     1              1         2052     0
         PetscRandom     1              1          648     0
  Spectral Transform     1              1          840     0
       Basis Vectors     1              1         9656     0
              Region     1              1          648     0
       Direct solver     1              1        20600     0
       Krylov Solver     1              1         1160     0
      Preconditioner     1              1         1096     0
========================================================================================================================
Average time to get PetscTime(): 1.19209e-07
Average time for MPI_Barrier(): 0.000147009
Average time for zero size MPI_Send(): 8.14795e-05
#PETSc Option Table entries:
-eps_max_it 2000
-eps_nev 1
-f1 a2
-f2 b2
-log_summary
-mat_mumps_icntl_14 200
-st_ksp_type preonly
-st_pc_factor_mat_solver_package mumps
-st_pc_type lu
-st_type sinvert
#End of PETSc Option Table entries
Compiled with FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 sizeof(PetscInt) 4
Configure options: PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
-----------------------------------------
Libraries compiled on Mon May 18 15:07:10 2015 on earth.ceas.iisc.ernet.in 
Machine characteristics: Linux-2.6.32-279.14.1.el6.x86_64-x86_64-with-centos-6.3-Final
Using PETSc directory: /cluster/share/venkatesh/petsc-3.5.3
Using PETSc arch: linux-gnu
-----------------------------------------

Using C compiler: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpicc  -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpif90  -fPIC  -Wall -Wno-unused-variable -ffree-line-length-0 -O  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/include -I/cluster/share/venkatesh/petsc-3.5.3/include -I/cluster/share/venkatesh/petsc-3.5.3/include -I/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/include
-----------------------------------------

Using C linker: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpicc
Using Fortran linker: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpif90
Using libraries: -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lpetsc -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lflapack -lfblas -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmpichf90 -lgfortran -lm -lmpichcxx -lstdc++ -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmpichcxx -lstdc++ -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -ldl -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl  
-------------- next part --------------
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex7 on a linux-gnu named compute-0-0.local with 2 processors, by venkatesh Mon May 18 23:59:48 2015
Using Petsc Release Version 3.5.3, Jan, 31, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.291e+02      1.00000   2.291e+02
Objects:              6.200e+01      1.03333   6.100e+01
Flops:                7.555e+09      1.06786   7.315e+09  1.463e+10
Flops/sec:            3.298e+07      1.06786   3.193e+07  6.386e+07
MPI Messages:         7.633e+03      1.00000   7.633e+03  1.527e+04
MPI Message Lengths:  2.313e+08      1.00000   3.030e+04  4.625e+08
MPI Reductions:       2.195e+04      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.2909e+02 100.0%  1.4629e+10 100.0%  1.527e+04 100.0%  3.030e+04      100.0%  2.195e+04 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult             7628 1.0 7.5923e-01 3.1 4.87e+0871.0 4.0e+00 2.4e+04 0.0e+00  0  3  0  0  0   0  3  0  0  0   650
MatSolve            7624 1.0 2.1264e+02 1.0 0.00e+00 0.0 1.5e+04 3.0e+04 7.6e+03 93  0100 98 35  93  0100 98 35     0
MatLUFactorSym         1 1.0 9.4827e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 9.8847e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  4  0  0  0  0   4  0  0  0  0     0
MatAssemblyBegin       2 1.0 1.5272e-0268.8 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         2 1.0 1.6832e-02 1.0 0.00e+00 0.0 4.0e+00 3.0e+03 1.6e+01  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 2.8610e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.1611e-04 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLoad                2 1.0 1.1695e-01 1.0 0.00e+00 0.0 1.0e+01 8.6e+05 2.6e+01  0  0  0  2  0   0  0  0  2  0     0
VecNorm                4 1.0 8.0910e-0341.9 5.76e+04 1.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0    14
VecCopy              955 1.0 4.1749e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              7629 1.0 6.4457e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                2 1.0 1.8835e-05 1.1 2.88e+04 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3058
VecScatterBegin    22876 1.0 5.9344e-01 1.4 0.00e+00 0.0 1.5e+04 3.0e+04 7.6e+03  0  0100 98 35   0  0100 98 35     0
VecScatterEnd      15252 1.0 2.2672e-01 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
EPSSetUp               1 1.0 1.0835e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+01  5  0  0  0  0   5  0  0  0  0     0
EPSSolve               1 1.0 2.2895e+02 1.0 7.54e+09 1.1 1.5e+04 3.0e+04 2.2e+04100100100 98100 100100100 98100    64
STSetUp                1 1.0 1.0834e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.3e+01  5  0  0  0  0   5  0  0  0  0     0
STApply             7624 1.0 2.1335e+02 1.0 4.76e+08 0.0 1.5e+04 3.0e+04 7.6e+03 93  3100 98 35  93  3100 98 35     2
STMatSolve          7624 1.0 2.1307e+02 1.0 0.00e+00 0.0 1.5e+04 3.0e+04 7.6e+03 93  0100 98 35  93  0100 98 35     0
BVCopy               954 1.0 7.0782e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BVMult             15224 1.0 1.9998e+00 1.0 4.28e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1 59  0  0  0   1 59  0  0  0  4281
BVDot              14271 1.0 2.2638e+00 1.0 2.73e+09 1.0 0.0e+00 0.0e+00 1.4e+04  1 37  0  0 65   1 37  0  0 65  2414
BVOrthogonalize     7625 1.0 3.4605e+00 1.0 5.26e+09 1.0 0.0e+00 0.0e+00 1.4e+04  2 72  0  0 65   2 72  0  0 65  3040
BVScale             7625 1.0 5.1506e-02 1.2 5.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  2132
BVSetRandom            1 1.0 1.8215e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSSolve              952 1.0 2.1509e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSVectors            955 1.0 8.1818e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSOther              952 1.0 8.5477e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               1 1.0 1.1921e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            7624 1.0 2.1285e+02 1.0 0.00e+00 0.0 1.5e+04 3.0e+04 7.6e+03 93  0100 98 35  93  0100 98 35     0
PCSetUp                1 1.0 1.0834e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+01  5  0  0  0  0   5  0  0  0  0     0
PCApply             7624 1.0 2.1266e+02 1.0 0.00e+00 0.0 1.5e+04 3.0e+04 7.6e+03 93  0100 98 35  93  0100 98 35     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     3              2         1504     0
              Matrix    11             11     14200392     0
              Vector    23             23      1015008     0
      Vector Scatter     4              4        15536     0
           Index Set    13             13        48956     0
Eigenvalue Problem Solver     1              1         2052     0
         PetscRandom     1              1          648     0
  Spectral Transform     1              1          840     0
       Basis Vectors     1              1         9656     0
              Region     1              1          648     0
       Direct solver     1              1        20600     0
       Krylov Solver     1              1         1160     0
      Preconditioner     1              1         1096     0
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 2.24113e-05
Average time for zero size MPI_Send(): 1.14441e-05
#PETSc Option Table entries:
-eps_max_it 2000
-eps_nev 1
-f1 a2
-f2 b2
-log_summary
-mat_mumps_icntl_14 85
-st_ksp_type preonly
-st_pc_factor_mat_solver_package mumps
-st_pc_type lu
-st_type sinvert
#End of PETSc Option Table entries
Compiled with FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 sizeof(PetscInt) 4
Configure options: PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
-----------------------------------------
Libraries compiled on Mon May 18 15:07:10 2015 on earth.ceas.iisc.ernet.in 
Machine characteristics: Linux-2.6.32-279.14.1.el6.x86_64-x86_64-with-centos-6.3-Final
Using PETSc directory: /cluster/share/venkatesh/petsc-3.5.3
Using PETSc arch: linux-gnu
-----------------------------------------

Using C compiler: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpicc  -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpif90  -fPIC  -Wall -Wno-unused-variable -ffree-line-length-0 -O  ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/include -I/cluster/share/venkatesh/petsc-3.5.3/include -I/cluster/share/venkatesh/petsc-3.5.3/include -I/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/include
-----------------------------------------

Using C linker: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpicc
Using Fortran linker: /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpif90
Using libraries: -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lpetsc -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lflapack -lfblas -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmpichf90 -lgfortran -lm -lmpichcxx -lstdc++ -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmpichcxx -lstdc++ -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -L/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.6 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -ldl -Wl,-rpath,/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/lib -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl  
-----------------------------------------

From knepley at gmail.com  Mon May 18 08:46:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 18 May 2015 08:46:58 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
Message-ID: <CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>

On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi I have attached the performance logs for 2 jobs on different
> processors. I had to increase the workspace icntl(14) when I submit on more
> cores since it is failing with small value of icntl(14).
>
> 1. performance_log1.txt is run on 8 cores (option given
> -mat_mumps_icntl_14 200)
> 2. performance_log2.txt is run on 2 cores (option given
> -mat_mumps_icntl_14 85  )
>

1) Your number of iterates increased from 7600 to 9600, but that is a
relatively small effect

2) MUMPS is just taking a lot longer to do forward/backward solve. You
might try emailing
the list for them. However, I would bet that your system has enough
bandwidth for 2 procs
and not much more.

  Thanks,

     Matt


> Venkatesh
>
> On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi, Thanks for the information. I now increased the workspace by adding
>>> '-mat_mumps_icntl_14 100'
>>>
>>> It works. However, the problem is, if I submit in 1 core I get the
>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>> 3500secs.
>>>
>>
>> Send the output of -log_summary for all performance queries. Otherwise we
>> are just guessing.
>>
>>     Matt
>>
>> My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>
>>> Kindly let me know.
>>>
>>> Venkatesh
>>>
>>>
>>>
>>> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>> david.knezevic at akselos.com> wrote:
>>>
>>>> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>
>>>>> A and B are of sizes 3600x3600
>>>>>
>>>>> I run with this command :
>>>>>
>>>>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>
>>>>> I get this error: (I get result only when I give 1 or 2 processors)
>>>>> Reading COMPLEX matrices from binary files...
>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>> [0]PETSC ERROR: Error in external library!
>>>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>>
>>>>
>>>>
>>>> The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>>>> In this case you have INFO(1)=-9, which is explained in the manual as:
>>>>
>>>> "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>> positive, then the number of entries that are missing in S at the moment
>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>> ICNTL(23) should be increased."
>>>>
>>>> This says that you should use ICTNL(14) to increase the working space
>>>> size:
>>>>
>>>> "ICNTL(14) is accessed by the host both during the analysis and the
>>>> factorization phases. It corresponds to the percentage increase in the
>>>> estimated working space. When significant extra fill-in is caused by
>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>
>>>> So, for example, you can avoid this error via the following command
>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/ca35e97b/attachment.html>

From balay at mcs.anl.gov  Mon May 18 10:22:31 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 18 May 2015 10:22:31 -0500
Subject: [petsc-users] Reg : Configuring PETSc on Cygwin
In-Reply-To: <AA7F30F772C09F47AE19F45AD9B68B785515AA2D@POCITMSEXMB06.LntUniverse.com>
References: <AA7F30F772C09F47AE19F45AD9B68B785515AA2D@POCITMSEXMB06.LntUniverse.com>
Message-ID: <alpine.LFD.2.11.1505181017460.18494@asterix>


1. please post issues to either petsc-users [public list] or
petsc-maint [private list] - but not both. And please do not bcc:
e-mails to the list. They will not get posed.

2. configure.log attached is a link file - the actual file shold
be in PETSC_DIR/PETSC_ARCH/conf/

3.when specifying PATHs to configure - use cygwin path notation. i.e

--download-fblaslapack=/cygdrive/D/Software/Cygwin/package/fblaslapack-3.4.2.tar.gz

4. you can install blas/lapack from cygwin [liblapack-devel] - and use it. Similarly with openmpi

5. You should not need/use  '--with-batch --ignore-cygwin-link' options with cygwin/gnu compilers.

Satish

On Mon, 18 May 2015, Manish Kumar K wrote:

> Dear PETSc Team,
> 
> 
>           I am configuring and trying to install PETSc on Cygwin shell.
> While invoking following commands in shell
> 
> Command : ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-batch --ignore-cygwin-link --with-mpi=0 --download-fblaslapack=D:/Software/Cygwin/package/fblaslapack-3.4.2.tar.gz
> 
> Following is the error I get :
> 
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> ===============================================================================                               Trying to download D:/Software/Cygwin/package/fblaslapack-3.4.2.tar.gz for FBLASLAPACK            ===============================================================================                                                                                                                                 *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Unable to download package fblaslapack from: D:/Software/Cygwin/package/fblaslapack-3.4.2.tar.gz
> * If URL specified manually - perhaps there is a typo?
> * If your network is disconnected - please reconnect and rerun ./configure
> * Or perhaps you have a firewall blocking the download
> * Alternatively, you can download the above URL manually, to /yourselectedlocation/fblaslapack-3.4.2.tar.gz
>   and use the configure option:
>   --download-fblaslapack=/yourselectedlocation/fblaslapack-3.4.2.tar.gz
> 
> I have tried the below also for typo check
> 
> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-batch --ignore-cygwin-link --with-mpi=0 --download-fblaslapack="D:/Software/Cygwin/package/fblaslapack-3.4.2.tar.gz"
> 
> Still the same error
> 
> I have checked my network connection is fine, firewall as well . is a typo I am doing in my commands inputs?
> I have attached a configure log  file for your reference and snapshot of Cygwin shell with above error
> Please help me out in this it would be great for me and my clients.
> 
> 
> 
> Best Regards
> Manish K
> 
> 
> 
> 
> L&T Technology Services Ltd
> 
> www.LntTechservices.com<http://www.lnttechservices.com/>
> 
> This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
> 


From venkateshgk.j at gmail.com  Mon May 18 11:14:26 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Mon, 18 May 2015 21:44:26 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
Message-ID: <CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>

Hi,
I have emailed the mumps-user list.
Actually the cluster has 8 nodes with 16 cores, and other codes scale well.
I wanted to ask if this job takes much time, then if I submit on more
cores, I have to increase the icntl(14).. which would again take long time.

So is there another way ?

cheers,
Venkatesh

On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi I have attached the performance logs for 2 jobs on different
>> processors. I had to increase the workspace icntl(14) when I submit on more
>> cores since it is failing with small value of icntl(14).
>>
>> 1. performance_log1.txt is run on 8 cores (option given
>> -mat_mumps_icntl_14 200)
>> 2. performance_log2.txt is run on 2 cores (option given
>> -mat_mumps_icntl_14 85  )
>>
>
> 1) Your number of iterates increased from 7600 to 9600, but that is a
> relatively small effect
>
> 2) MUMPS is just taking a lot longer to do forward/backward solve. You
> might try emailing
> the list for them. However, I would bet that your system has enough
> bandwidth for 2 procs
> and not much more.
>
>   Thanks,
>
>      Matt
>
>
>> Venkatesh
>>
>> On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi, Thanks for the information. I now increased the workspace by adding
>>>> '-mat_mumps_icntl_14 100'
>>>>
>>>> It works. However, the problem is, if I submit in 1 core I get the
>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>> 3500secs.
>>>>
>>>
>>> Send the output of -log_summary for all performance queries. Otherwise
>>> we are just guessing.
>>>
>>>     Matt
>>>
>>> My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>
>>>> Kindly let me know.
>>>>
>>>> Venkatesh
>>>>
>>>>
>>>>
>>>> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>> david.knezevic at akselos.com> wrote:
>>>>
>>>>> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>
>>>>>> A and B are of sizes 3600x3600
>>>>>>
>>>>>> I run with this command :
>>>>>>
>>>>>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>
>>>>>> I get this error: (I get result only when I give 1 or 2 processors)
>>>>>> Reading COMPLEX matrices from binary files...
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>> ------------------------------------
>>>>>> [0]PETSC ERROR: Error in external library!
>>>>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>>>
>>>>>
>>>>>
>>>>> The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>>>>> In this case you have INFO(1)=-9, which is explained in the manual as:
>>>>>
>>>>> "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>>> positive, then the number of entries that are missing in S at the moment
>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>> ICNTL(23) should be increased."
>>>>>
>>>>> This says that you should use ICTNL(14) to increase the working space
>>>>> size:
>>>>>
>>>>> "ICNTL(14) is accessed by the host both during the analysis and the
>>>>> factorization phases. It corresponds to the percentage increase in the
>>>>> estimated working space. When significant extra fill-in is caused by
>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>
>>>>> So, for example, you can avoid this error via the following command
>>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/c25b70b9/attachment.html>

From hzhang at mcs.anl.gov  Mon May 18 11:41:19 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Mon, 18 May 2015 11:41:19 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
Message-ID: <CAGCphBvZG-kcY4CKTatjTS1j9_ixWMr0Q_v8n3cu4Rtj6ttY-g@mail.gmail.com>

venkatesh:
The differences are in MatSolve() as Matt said.
The questions are

1. why np=2 calls 7624 MatSolve(), while np=8 calls 9600 times? With
slepc/sinvert/lu, would both runs use same algorithm?

2. with np=2, each MatSolve takes 0.027 sec, while np=8 takes 0.065 sec.
Would MatSolve be scalable?
The matrix factors for np=2 and 8 might be very different.

We would like to know what mumps' developer say about it.

Hong

Hi,
> I have emailed the mumps-user list.
> Actually the cluster has 8 nodes with 16 cores, and other codes scale
> well.
> I wanted to ask if this job takes much time, then if I submit on more
> cores, I have to increase the icntl(14).. which would again take long time.
>
> So is there another way ?
>
> cheers,
> Venkatesh
>
> On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi I have attached the performance logs for 2 jobs on different
>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>> cores since it is failing with small value of icntl(14).
>>>
>>> 1. performance_log1.txt is run on 8 cores (option given
>>> -mat_mumps_icntl_14 200)
>>> 2. performance_log2.txt is run on 2 cores (option given
>>> -mat_mumps_icntl_14 85  )
>>>
>>
>> 1) Your number of iterates increased from 7600 to 9600, but that is a
>> relatively small effect
>>
>> 2) MUMPS is just taking a lot longer to do forward/backward solve. You
>> might try emailing
>> the list for them. However, I would bet that your system has enough
>> bandwidth for 2 procs
>> and not much more.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Venkatesh
>>>
>>> On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, Thanks for the information. I now increased the workspace by
>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>
>>>>> It works. However, the problem is, if I submit in 1 core I get the
>>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>>> 3500secs.
>>>>>
>>>>
>>>> Send the output of -log_summary for all performance queries. Otherwise
>>>> we are just guessing.
>>>>
>>>>     Matt
>>>>
>>>> My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>
>>>>> Kindly let me know.
>>>>>
>>>>> Venkatesh
>>>>>
>>>>>
>>>>>
>>>>> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>> david.knezevic at akselos.com> wrote:
>>>>>
>>>>>> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>
>>>>>>> A and B are of sizes 3600x3600
>>>>>>>
>>>>>>> I run with this command :
>>>>>>>
>>>>>>> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>
>>>>>>> I get this error: (I get result only when I give 1 or 2 processors)
>>>>>>> Reading COMPLEX matrices from binary files...
>>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>>>> ------------------------------------
>>>>>>> [0]PETSC ERROR: Error in external library!
>>>>>>> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>
>>>>>>
>>>>>>
>>>>>> The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>>>>>> In this case you have INFO(1)=-9, which is explained in the manual as:
>>>>>>
>>>>>> "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>>>> positive, then the number of entries that are missing in S at the moment
>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>> ICNTL(23) should be increased."
>>>>>>
>>>>>> This says that you should use ICTNL(14) to increase the working space
>>>>>> size:
>>>>>>
>>>>>> "ICNTL(14) is accessed by the host both during the analysis and the
>>>>>> factorization phases. It corresponds to the percentage increase in the
>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>
>>>>>> So, for example, you can avoid this error via the following command
>>>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/98168847/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon May 18 12:51:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 18 May 2015 12:51:28 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
Message-ID: <539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>


   Run the streams benchmark on this system and send the results. http://www.mcs.anl.gov/petsc/documentation/faq.html#computers


> On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> 
> Hi,
> I have emailed the mumps-user list. 
> Actually the cluster has 8 nodes with 16 cores, and other codes scale well. 
> I wanted to ask if this job takes much time, then if I submit on more cores, I have to increase the icntl(14).. which would again take long time. 
> 
> So is there another way ?
> 
> cheers,
> Venkatesh
> 
> On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> Hi I have attached the performance logs for 2 jobs on different processors. I had to increase the workspace icntl(14) when I submit on more cores since it is failing with small value of icntl(14).
> 
> 1. performance_log1.txt is run on 8 cores (option given -mat_mumps_icntl_14 200)
> 2. performance_log2.txt is run on 2 cores (option given -mat_mumps_icntl_14 85  )
> 
> 1) Your number of iterates increased from 7600 to 9600, but that is a relatively small effect
> 
> 2) MUMPS is just taking a lot longer to do forward/backward solve. You might try emailing
> the list for them. However, I would bet that your system has enough bandwidth for 2 procs
> and not much more.
> 
>   Thanks,
> 
>      Matt
>  
> Venkatesh
> 
> On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> Hi, Thanks for the information. I now increased the workspace by adding '-mat_mumps_icntl_14 100' 
> 
> It works. However, the problem is, if I submit in 1 core I get the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes 3500secs.
> 
> Send the output of -log_summary for all performance queries. Otherwise we are just guessing.
> 
>     Matt
> 
> My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
> 
> Kindly let me know.
> 
> Venkatesh
> 
> 
> 
> On Sat, May 16, 2015 at 7:10 PM, David Knezevic <david.knezevic at akselos.com> wrote:
> On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> Hi,
> I am trying to solving AX=lambda BX eigenvalue problem.
> 
> A and B are of sizes 3600x3600
> 
> I run with this command :
> 
> 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps'
> 
> I get this error: (I get result only when I give 1 or 2 processors)
> Reading COMPLEX matrices from binary files...
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Error in external library!
> [0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=2024
> 
> 
> The MUMPS error types are described in Chapter 7 of the MUMPS manual. In this case you have INFO(1)=-9, which is explained in the manual as:
> 
> "?9 Main internal real/complex workarray S too small. If INFO(2) is positive, then the number of entries that are missing in S at the moment when the error is raised is available in INFO(2). If INFO(2) is negative, then its absolute value should be multiplied by 1 million. If an error ?9 occurs, the user should increase the value of ICNTL(14) before calling the factorization (JOB=2) again, except if ICNTL(23) is provided, in which case ICNTL(23) should be increased."
> 
> This says that you should use ICTNL(14) to increase the working space size:
> 
> "ICNTL(14) is accessed by the host both during the analysis and the factorization phases. It corresponds to the percentage increase in the estimated working space. When significant extra fill-in is caused by numerical pivoting, increasing ICNTL(14) may help. Except in special cases, the default value is 20 (which corresponds to a 20 % increase)."
> 
> So, for example, you can avoid this error via the following command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we allow a 30% increase in the workspace instead of the default 20%.
> 
> David
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From gbisht at lbl.gov  Mon May 18 13:35:02 2015
From: gbisht at lbl.gov (Gautam Bisht)
Date: Mon, 18 May 2015 11:35:02 -0700
Subject: [petsc-users] Installation error: C++ preprocessor
Message-ID: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>

Hi,

While trying to install v3.5.3 with intel 15.0.0 on a linux machine, I'm
getting an error related to C++ preprocessor. Any help to resolve the issue
would be greatly appreciated.

Attached below is the logfile.

Thanks,
-Gautam.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/06e66d76/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 3483849 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/06e66d76/attachment-0001.obj>

From knepley at gmail.com  Mon May 18 14:18:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 18 May 2015 14:18:10 -0500
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
Message-ID: <CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>

On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov> wrote:

> Hi,
>
> While trying to install v3.5.3 with intel 15.0.0 on a linux machine, I'm
> getting an error related to C++ preprocessor. Any help to resolve the issue
> would be greatly appreciated.
>
> Attached below is the logfile.
>

The HDF5 configure is failing because it says your preprocessor does not
work. We would need
to see

  /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log

  Thanks,

    Matt


> Thanks,
> -Gautam.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/ce5a6116/attachment.html>

From balay at mcs.anl.gov  Mon May 18 14:18:57 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 18 May 2015 14:18:57 -0500
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505181418310.18494@asterix>

On Mon, 18 May 2015, Matthew Knepley wrote:

> On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov> wrote:
> 
> > Hi,
> >
> > While trying to install v3.5.3 with intel 15.0.0 on a linux machine, I'm
> > getting an error related to C++ preprocessor. Any help to resolve the issue
> > would be greatly appreciated.
> >
> > Attached below is the logfile.
> >
> 
> The HDF5 configure is failing because it says your preprocessor does not
> work. We would need
> to see
> 
>   /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log

I think the issue is - hdf5 configure is looking for c++ compiler -
but PETSc configure is not providing this info to hdf5 configure.

Satish

From gbisht at lbl.gov  Mon May 18 14:28:06 2015
From: gbisht at lbl.gov (Gautam Bisht)
Date: Mon, 18 May 2015 12:28:06 -0700
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <alpine.LFD.2.11.1505181418310.18494@asterix>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
	<alpine.LFD.2.11.1505181418310.18494@asterix>
Message-ID: <CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>

Attached below is the config.log for HDF5.

-Gautam.

On Mon, May 18, 2015 at 12:18 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Mon, 18 May 2015, Matthew Knepley wrote:
>
> > On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov> wrote:
> >
> > > Hi,
> > >
> > > While trying to install v3.5.3 with intel 15.0.0 on a linux machine,
> I'm
> > > getting an error related to C++ preprocessor. Any help to resolve the
> issue
> > > would be greatly appreciated.
> > >
> > > Attached below is the logfile.
> > >
> >
> > The HDF5 configure is failing because it says your preprocessor does not
> > work. We would need
> > to see
> >
> >
>  /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log
>
> I think the issue is - hdf5 configure is looking for c++ compiler -
> but PETSc configure is not providing this info to hdf5 configure.
>
> Satish
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/dec326a5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hdf5-1.8.10-patch1.1.config.log
Type: application/octet-stream
Size: 82302 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150518/dec326a5/attachment-0001.obj>

From balay at mcs.anl.gov  Mon May 18 14:33:28 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Mon, 18 May 2015 14:33:28 -0500
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
	<alpine.LFD.2.11.1505181418310.18494@asterix>
	<CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505181430440.18494@asterix>

Can you try using petsc development branch 'master' and see if
it works? It should also add in the c++ compiler to hdf5 configure..

Wrt petsc-3.5 - perhaps the attached patch will work..

Satish

On Mon, 18 May 2015, Gautam Bisht wrote:

> Attached below is the config.log for HDF5.
> 
> -Gautam.
> 
> On Mon, May 18, 2015 at 12:18 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Mon, 18 May 2015, Matthew Knepley wrote:
> >
> > > On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov> wrote:
> > >
> > > > Hi,
> > > >
> > > > While trying to install v3.5.3 with intel 15.0.0 on a linux machine,
> > I'm
> > > > getting an error related to C++ preprocessor. Any help to resolve the
> > issue
> > > > would be greatly appreciated.
> > > >
> > > > Attached below is the logfile.
> > > >
> > >
> > > The HDF5 configure is failing because it says your preprocessor does not
> > > work. We would need
> > > to see
> > >
> > >
> >  /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log
> >
> > I think the issue is - hdf5 configure is looking for c++ compiler -
> > but PETSc configure is not providing this info to hdf5 configure.
> >
> > Satish
> >
> 
-------------- next part --------------
diff --git a/config/BuildSystem/config/packages/hdf5.py b/config/BuildSystem/config/packages/hdf5.py
index 36ad5ad..944a28a 100644
--- a/config/BuildSystem/config/packages/hdf5.py
+++ b/config/BuildSystem/config/packages/hdf5.py
@@ -40,6 +40,12 @@ class Configure(config.package.Package):
     args.append('CFLAGS="'+self.framework.getCompilerFlags()+'"')
     self.framework.popLanguage()
     args.append('--enable-parallel')
+    if hasattr(self.compilers, 'CXX'):
+      self.framework.pushLanguage('Cxx')
+      args.append('CXX="'+self.framework.getCompiler()+' '+self.framework.getCompilerFlags()+'"')
+      self.framework.popLanguage()
+    else:
+      raise RuntimeError('Error: HDF5 requires C++ compiler. None specified')
     if hasattr(self.compilers, 'FC'):
       self.setCompilers.pushLanguage('FC')
       args.append('--enable-fortran')

From alpkalpalp at gmail.com  Mon May 18 17:07:02 2015
From: alpkalpalp at gmail.com (Alp Kalpalp)
Date: Tue, 19 May 2015 01:07:02 +0300
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
In-Reply-To: <A4C36052-A95A-4804-A4C9-DF05E1481CAE@dsic.upv.es>
References: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
	<CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>
	<CABzT7hN3yaoCYyNK46QBzdKeBHNyGR6PneWEqUJUuwmBJ2SjBg@mail.gmail.com>
	<A4C36052-A95A-4804-A4C9-DF05E1481CAE@dsic.upv.es>
Message-ID: <CALi504SQ=Ch7A2Z2J=eswVg_gNd1UquaVNt6Ty=Mm_sEePcDtg@mail.gmail.com>

Hi,

While reading the link you gave, I have seen a strange sentence;

"Otherwise you will get strange crashes."

can someone explain it why?

On Sun, May 17, 2015 at 12:16 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> http://www.mcs.anl.gov/petsc/documentation/faq.html#with-64-bit-indices
>
>
>
> El 17/05/2015, a las 09:02, venkatesh g escribi?:
>
> > Thank you. So the single eigenvector's size which is Nx1 is the number
> of unknowns and should be less than 2^31-1 so as to avoid configuring with
> '--with-64-bit-indices=1' ??
> >
> >
> > On Sat, May 16, 2015 at 8:05 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Sat, May 16, 2015 at 3:49 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
> > Hi,
> >
> > I am trying to solving AX=lambda BX
> >
> > A and B are of size 57000 x 57000
> >
> > Currently I solve my small matrices using -st_type sinvert and -eps_nev 1
> >
> > So I get 1 eigenvalue and eigenvector.
> >
> > So for the 57000x57000 large matrix problem, the unknown eigenvector is
> 57000x1
> >
> > Should I configure the petsc as '--with-64-bit-indices=1' for this
> problem ?
> >
> > It is not necessary.
> >
> >    Matt
> >
> > Is the unknowns being considered as the entire eigenspace 57000x57000 ?
> >
> > kindly let me know.
> >
> > Venkatesh
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/06e01a88/attachment.html>

From bsmith at mcs.anl.gov  Mon May 18 17:16:35 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 18 May 2015 17:16:35 -0500
Subject: [petsc-users] Large Matrix eigenvalue problem : configuration
In-Reply-To: <CALi504SQ=Ch7A2Z2J=eswVg_gNd1UquaVNt6Ty=Mm_sEePcDtg@mail.gmail.com>
References: <CABzT7hPYfWusGeiRotj_kNeg=Y8-Ek-kcSkMD+LeiPkF4LmvwQ@mail.gmail.com>
	<CAMYG4GkfQ1J0R1E2jBCTKGAnN9Z-asm-atZDy9Fx8pa19ossjw@mail.gmail.com>
	<CABzT7hN3yaoCYyNK46QBzdKeBHNyGR6PneWEqUJUuwmBJ2SjBg@mail.gmail.com>
	<A4C36052-A95A-4804-A4C9-DF05E1481CAE@dsic.upv.es>
	<CALi504SQ=Ch7A2Z2J=eswVg_gNd1UquaVNt6Ty=Mm_sEePcDtg@mail.gmail.com>
Message-ID: <73DEFB28-3F32-47C7-9F2C-A8BC8D11F413@mcs.anl.gov>


> On May 18, 2015, at 5:07 PM, Alp Kalpalp <alpkalpalp at gmail.com> wrote:
> 
> Hi,
> 
> While reading the link you gave, I have seen a strange sentence;
> 
> "Otherwise you will get strange crashes."
> 
> can someone explain it why?

   Integer overflow is not really handled well in software. So when you add two int numbers together (each less than the maximum value that can go into an int, but whose sum won't fit into an int) you can end up with a negative number or a much smaller positive number. The code just keeps on chugging along with these "incorrect" numbers (producing incorrect results but not necessarily crashing or providing any indication to the user that something is wrong). Eventually if you are lucky something gets so out of whack it crashes and so you know something is wrong. If you are unlucky it never crashes and you think the results are fine when they are not.

  Barry



> 
> On Sun, May 17, 2015 at 12:16 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> http://www.mcs.anl.gov/petsc/documentation/faq.html#with-64-bit-indices
> 
> 
> 
> El 17/05/2015, a las 09:02, venkatesh g escribi?:
> 
> > Thank you. So the single eigenvector's size which is Nx1 is the number of unknowns and should be less than 2^31-1 so as to avoid configuring with '--with-64-bit-indices=1' ??
> >
> >
> > On Sat, May 16, 2015 at 8:05 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Sat, May 16, 2015 at 3:49 AM, venkatesh g <venkateshgk.j at gmail.com> wrote:
> > Hi,
> >
> > I am trying to solving AX=lambda BX
> >
> > A and B are of size 57000 x 57000
> >
> > Currently I solve my small matrices using -st_type sinvert and -eps_nev 1
> >
> > So I get 1 eigenvalue and eigenvector.
> >
> > So for the 57000x57000 large matrix problem, the unknown eigenvector is 57000x1
> >
> > Should I configure the petsc as '--with-64-bit-indices=1' for this problem ?
> >
> > It is not necessary.
> >
> >    Matt
> >
> > Is the unknowns being considered as the entire eigenspace 57000x57000 ?
> >
> > kindly let me know.
> >
> > Venkatesh
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> 
> 


From venkateshgk.j at gmail.com  Tue May 19 01:04:48 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Tue, 19 May 2015 11:34:48 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
Message-ID: <CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>

Hi,

I have attached the log of the command which I gave in the master node:
make streams NPMAX=32

I dont know why it says 'It appears you have only 1 node'. But other codes
run in parallel with good scaling on 8 nodes.

Kindly let me know.

Venkatesh



On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Run the streams benchmark on this system and send the results.
> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>
>
> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
> >
> > Hi,
> > I have emailed the mumps-user list.
> > Actually the cluster has 8 nodes with 16 cores, and other codes scale
> well.
> > I wanted to ask if this job takes much time, then if I submit on more
> cores, I have to increase the icntl(14).. which would again take long time.
> >
> > So is there another way ?
> >
> > cheers,
> > Venkatesh
> >
> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
> > Hi I have attached the performance logs for 2 jobs on different
> processors. I had to increase the workspace icntl(14) when I submit on more
> cores since it is failing with small value of icntl(14).
> >
> > 1. performance_log1.txt is run on 8 cores (option given
> -mat_mumps_icntl_14 200)
> > 2. performance_log2.txt is run on 2 cores (option given
> -mat_mumps_icntl_14 85  )
> >
> > 1) Your number of iterates increased from 7600 to 9600, but that is a
> relatively small effect
> >
> > 2) MUMPS is just taking a lot longer to do forward/backward solve. You
> might try emailing
> > the list for them. However, I would bet that your system has enough
> bandwidth for 2 procs
> > and not much more.
> >
> >   Thanks,
> >
> >      Matt
> >
> > Venkatesh
> >
> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
> > Hi, Thanks for the information. I now increased the workspace by adding
> '-mat_mumps_icntl_14 100'
> >
> > It works. However, the problem is, if I submit in 1 core I get the
> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
> 3500secs.
> >
> > Send the output of -log_summary for all performance queries. Otherwise
> we are just guessing.
> >
> >     Matt
> >
> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
> >
> > Kindly let me know.
> >
> > Venkatesh
> >
> >
> >
> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
> > Hi,
> > I am trying to solving AX=lambda BX eigenvalue problem.
> >
> > A and B are of sizes 3600x3600
> >
> > I run with this command :
> >
> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps'
> >
> > I get this error: (I get result only when I give 1 or 2 processors)
> > Reading COMPLEX matrices from binary files...
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Error in external library!
> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
> phase: INFO(1)=-9, INFO(2)=2024
> >
> >
> > The MUMPS error types are described in Chapter 7 of the MUMPS manual. In
> this case you have INFO(1)=-9, which is explained in the manual as:
> >
> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
> positive, then the number of entries that are missing in S at the moment
> when the error is raised is available in INFO(2). If INFO(2) is negative,
> then its absolute value should be multiplied by 1 million. If an error ?9
> occurs, the user should increase the value of ICNTL(14) before calling the
> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
> ICNTL(23) should be increased."
> >
> > This says that you should use ICTNL(14) to increase the working space
> size:
> >
> > "ICNTL(14) is accessed by the host both during the analysis and the
> factorization phases. It corresponds to the percentage increase in the
> estimated working space. When significant extra fill-in is caused by
> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
> the default value is 20 (which corresponds to a 20 % increase)."
> >
> > So, for example, you can avoid this error via the following command line
> argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that we
> allow a 30% increase in the workspace instead of the default 20%.
> >
> > David
> >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/6f0eccf1/attachment-0001.html>
-------------- next part --------------
cd src/benchmarks/streams; /usr/bin/gmake  --no-print-directory streams
/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpicc -o MPIVersion.o -c -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O   -I/cluster/share/venkatesh/petsc-3.5.3/include -I/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/include    `pwd`/MPIVersion.c
/cluster/share/venkatesh/petsc-3.5.3/src/benchmarks/streams/MPIVersion.c: In function ?main?:
/cluster/share/venkatesh/petsc-3.5.3/src/benchmarks/streams/MPIVersion.c:99: warning: value computed is not used
/cluster/share/venkatesh/petsc-3.5.3/src/benchmarks/streams/MPIVersion.c:103: warning: value computed is not used
Number of MPI processes 1
Process 0 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:        6902.4288
Scale:       6686.1477
Add:         7267.3209
Triad:       7347.4177
Number of MPI processes 2
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17086.9319
Scale:      16523.1725
Add:        18397.7537
Triad:      18843.7471
Number of MPI processes 3
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18378.9967
Scale:      17876.0998
Add:        19955.7187
Triad:      20279.8005
Number of MPI processes 4
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18317.7102
Scale:      17908.8977
Add:        20086.3669
Triad:      20595.9138
Number of MPI processes 5
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18014.6277
Scale:      17749.5260
Add:        20221.8266
Triad:      20523.3457
Number of MPI processes 6
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17707.3799
Scale:      17312.7292
Add:        19866.6142
Triad:      20421.0646
Number of MPI processes 7
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18032.9462
Scale:      18227.9864
Add:        20153.5011
Triad:      20272.5224
Number of MPI processes 8
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18157.8195
Scale:      17795.5266
Add:        20279.1823
Triad:      20429.6992
Number of MPI processes 9
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17992.4195
Scale:      17895.6520
Add:        20261.2123
Triad:      20268.0128
Number of MPI processes 10
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       18032.9722
Scale:      17803.6405
Add:        20275.4183
Triad:      20331.5734
Number of MPI processes 11
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17979.6366
Scale:      17938.5844
Add:        20025.9127
Triad:      20264.9269
Number of MPI processes 12
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17999.2694
Scale:      17836.7113
Add:        20215.8741
Triad:      20148.6332
Number of MPI processes 13
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17843.2142
Scale:      17913.6416
Add:        20051.1341
Triad:      20248.3982
Number of MPI processes 14
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17726.6043
Scale:      17846.1749
Add:        20133.6683
Triad:      20116.7300
Number of MPI processes 15
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17748.3574
Scale:      17730.9431
Add:        20131.9446
Triad:      20140.4377
Number of MPI processes 16
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17851.9776
Scale:      17612.7354
Add:        20050.5608
Triad:      20294.3173
Number of MPI processes 17
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17736.6966
Scale:      17799.4394
Add:        20238.6540
Triad:      19922.1132
Number of MPI processes 18
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17713.9050
Scale:      17704.2554
Add:        19988.1965
Triad:      20111.7783
Number of MPI processes 19
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17845.2428
Scale:      17716.9512
Add:        19960.8970
Triad:      19996.5972
Number of MPI processes 20
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17679.7657
Scale:      17788.3173
Add:        19918.5972
Triad:      20140.1278
Number of MPI processes 21
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17729.0197
Scale:      17551.8412
Add:        19939.2570
Triad:      20018.7010
Number of MPI processes 22
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17718.9268
Scale:      17646.6164
Add:        19958.9034
Triad:      20019.6979
Number of MPI processes 23
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17726.1052
Scale:      17586.0347
Add:        19981.6659
Triad:      20129.2205
Number of MPI processes 24
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17679.1036
Scale:      17557.5055
Add:        20004.1105
Triad:      20058.5391
Number of MPI processes 25
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17701.5456
Scale:      17660.2441
Add:        19982.5803
Triad:      20008.9784
Number of MPI processes 26
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17689.5153
Scale:      17690.2272
Add:        19992.6007
Triad:      20009.8738
Number of MPI processes 27
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17679.5819
Scale:      17723.8157
Add:        20062.3149
Triad:      20088.0081
Number of MPI processes 28
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Process 27 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17713.3862
Scale:      17653.8698
Add:        20033.4436
Triad:      20090.3366
Number of MPI processes 29
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Process 27 earth.ceas.iisc.ernet.in
Process 28 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17715.5523
Scale:      17658.6908
Add:        19999.5113
Triad:      20055.8129
Number of MPI processes 30
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Process 27 earth.ceas.iisc.ernet.in
Process 28 earth.ceas.iisc.ernet.in
Process 29 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17706.5495
Scale:      17632.8140
Add:        20064.5500
Triad:      20080.0843
Number of MPI processes 31
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Process 27 earth.ceas.iisc.ernet.in
Process 28 earth.ceas.iisc.ernet.in
Process 29 earth.ceas.iisc.ernet.in
Process 30 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17729.5745
Scale:      17674.5220
Add:        20044.3250
Triad:      20092.4184
Number of MPI processes 32
Process 0 earth.ceas.iisc.ernet.in
Process 1 earth.ceas.iisc.ernet.in
Process 2 earth.ceas.iisc.ernet.in
Process 3 earth.ceas.iisc.ernet.in
Process 4 earth.ceas.iisc.ernet.in
Process 5 earth.ceas.iisc.ernet.in
Process 6 earth.ceas.iisc.ernet.in
Process 7 earth.ceas.iisc.ernet.in
Process 8 earth.ceas.iisc.ernet.in
Process 9 earth.ceas.iisc.ernet.in
Process 10 earth.ceas.iisc.ernet.in
Process 11 earth.ceas.iisc.ernet.in
Process 12 earth.ceas.iisc.ernet.in
Process 13 earth.ceas.iisc.ernet.in
Process 14 earth.ceas.iisc.ernet.in
Process 15 earth.ceas.iisc.ernet.in
Process 16 earth.ceas.iisc.ernet.in
Process 17 earth.ceas.iisc.ernet.in
Process 18 earth.ceas.iisc.ernet.in
Process 19 earth.ceas.iisc.ernet.in
Process 20 earth.ceas.iisc.ernet.in
Process 21 earth.ceas.iisc.ernet.in
Process 22 earth.ceas.iisc.ernet.in
Process 23 earth.ceas.iisc.ernet.in
Process 24 earth.ceas.iisc.ernet.in
Process 25 earth.ceas.iisc.ernet.in
Process 26 earth.ceas.iisc.ernet.in
Process 27 earth.ceas.iisc.ernet.in
Process 28 earth.ceas.iisc.ernet.in
Process 29 earth.ceas.iisc.ernet.in
Process 30 earth.ceas.iisc.ernet.in
Process 31 earth.ceas.iisc.ernet.in
Function      Rate (MB/s) 
Copy:       17731.4969
Scale:      17660.6923
Add:        20066.4112
Triad:      20130.9258
------------------------------------------------
np  speedup
1 1.0
2 2.56
3 2.76
4 2.8
5 2.79
6 2.78
7 2.76
8 2.78
9 2.76
10 2.77
11 2.76
12 2.74
13 2.76
14 2.74
15 2.74
16 2.76
17 2.71
18 2.74
19 2.72
20 2.74
21 2.72
22 2.72
23 2.74
24 2.73
25 2.72
26 2.72
27 2.73
28 2.73
29 2.73
30 2.73
31 2.73
32 2.74
Estimation of possible speedup of MPI programs based on Streams benchmark.
It appears you have 1 node(s)
Unable to open matplotlib to plot speedup
Unable to open matplotlib to plot speedup

From knepley at gmail.com  Tue May 19 04:32:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 May 2015 04:32:39 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
Message-ID: <CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>

On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
>
> I have attached the log of the command which I gave in the master node:
> make streams NPMAX=32
>
> I dont know why it says 'It appears you have only 1 node'. But other codes
> run in parallel with good scaling on 8 nodes.
>

If you look at the STREAMS numbers, you can see that your system is only
able to support about 2 cores with the
available memory bandwidth. Thus for bandwidth constrained operations
(almost everything in sparse linear algebra
and solvers), your speedup will not be bigger than 2.

Other codes may do well on this machine, but they would be compute
constrained, using things like DGEMM.

  Thanks,

     Matt


> Kindly let me know.
>
> Venkatesh
>
>
>
> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>    Run the streams benchmark on this system and send the results.
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>
>>
>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>> >
>> > Hi,
>> > I have emailed the mumps-user list.
>> > Actually the cluster has 8 nodes with 16 cores, and other codes scale
>> well.
>> > I wanted to ask if this job takes much time, then if I submit on more
>> cores, I have to increase the icntl(14).. which would again take long time.
>> >
>> > So is there another way ?
>> >
>> > cheers,
>> > Venkatesh
>> >
>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>> > Hi I have attached the performance logs for 2 jobs on different
>> processors. I had to increase the workspace icntl(14) when I submit on more
>> cores since it is failing with small value of icntl(14).
>> >
>> > 1. performance_log1.txt is run on 8 cores (option given
>> -mat_mumps_icntl_14 200)
>> > 2. performance_log2.txt is run on 2 cores (option given
>> -mat_mumps_icntl_14 85  )
>> >
>> > 1) Your number of iterates increased from 7600 to 9600, but that is a
>> relatively small effect
>> >
>> > 2) MUMPS is just taking a lot longer to do forward/backward solve. You
>> might try emailing
>> > the list for them. However, I would bet that your system has enough
>> bandwidth for 2 procs
>> > and not much more.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > Venkatesh
>> >
>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>> > Hi, Thanks for the information. I now increased the workspace by adding
>> '-mat_mumps_icntl_14 100'
>> >
>> > It works. However, the problem is, if I submit in 1 core I get the
>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>> 3500secs.
>> >
>> > Send the output of -log_summary for all performance queries. Otherwise
>> we are just guessing.
>> >
>> >     Matt
>> >
>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>> >
>> > Kindly let me know.
>> >
>> > Venkatesh
>> >
>> >
>> >
>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>> david.knezevic at akselos.com> wrote:
>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>> > Hi,
>> > I am trying to solving AX=lambda BX eigenvalue problem.
>> >
>> > A and B are of sizes 3600x3600
>> >
>> > I run with this command :
>> >
>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package mumps'
>> >
>> > I get this error: (I get result only when I give 1 or 2 processors)
>> > Reading COMPLEX matrices from binary files...
>> > [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> > [0]PETSC ERROR: Error in external library!
>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: INFO(1)=-9, INFO(2)=2024
>> >
>> >
>> > The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>> In this case you have INFO(1)=-9, which is explained in the manual as:
>> >
>> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
>> positive, then the number of entries that are missing in S at the moment
>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>> then its absolute value should be multiplied by 1 million. If an error ?9
>> occurs, the user should increase the value of ICNTL(14) before calling the
>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>> ICNTL(23) should be increased."
>> >
>> > This says that you should use ICTNL(14) to increase the working space
>> size:
>> >
>> > "ICNTL(14) is accessed by the host both during the analysis and the
>> factorization phases. It corresponds to the percentage increase in the
>> estimated working space. When significant extra fill-in is caused by
>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>> the default value is 20 (which corresponds to a 20 % increase)."
>> >
>> > So, for example, you can avoid this error via the following command
>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>> we allow a 30% increase in the workspace instead of the default 20%.
>> >
>> > David
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/9b56b0de/attachment.html>

From mlohry at princeton.edu  Tue May 19 10:14:32 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Tue, 19 May 2015 11:14:32 -0400
Subject: [petsc-users] DMPlex examples in parallel/3D?
Message-ID: <555B5358.5070501@princeton.edu>

I'm interested in trying DMPlex for a parallel 3D unstructured DG 
solver.  Is it appropriate for this, and is it considerably more work 
than for the simple DMDACreate3D examples? The manual example seems 
fairly clear on how to set up a mesh in serial, but does this play 
nicely with METIS for partitioning, and DMDA for parallel 
arrays/communication? I'm typically using CGNS files, so is 
DMPlexCreateCGNS working in parallel and appropriate for this?

Any examples/guidance related to this would be appreciated.

-Mark

From knepley at gmail.com  Tue May 19 10:24:16 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 May 2015 10:24:16 -0500
Subject: [petsc-users] DMPlex examples in parallel/3D?
In-Reply-To: <555B5358.5070501@princeton.edu>
References: <555B5358.5070501@princeton.edu>
Message-ID: <CAMYG4GnKx2D+cc8+ELDA=5Va-Po=XLSkdCZzC9Jm+NdAE0Cg5w@mail.gmail.com>

On Tue, May 19, 2015 at 10:14 AM, Mark Lohry <mlohry at princeton.edu> wrote:

> I'm interested in trying DMPlex for a parallel 3D unstructured DG solver.
> Is it appropriate for this, and is it considerably more work than for the
> simple DMDACreate3D examples? The manual example seems fairly clear on how
> to set up a mesh in serial, but does this play nicely with METIS for
> partitioning, and DMDA for parallel arrays/communication? I'm typically
> using CGNS files, so is DMPlexCreateCGNS working in parallel and
> appropriate for this?
>

Plex can certainly handle 3D unstructured meshes, read from CGNS, and can
partition them and repartition them.

You can easily get started by just calling DMPlexDistribute() on a serial
mesh.

To use it to handle fields, you construct a PetscSection that models your
data layout for the DG field. This just means
tell it how many dofs sit on each cell, face, edge, and vertex. Then it can
handle all the parallel communication for you.
Note that you need to specify what kind of adjacency you have for it to
automatically construct the comm pattern:


http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexSetAdjacencyUseClosure.html

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMPlexSetAdjacencyUseCone.html#DMPlexSetAdjacencyUseCone



> Any examples/guidance related to this would be appreciated.


I do cG FEM in SNES ex12 and ex62.

  Thanks,

     Matt

>
> -Mark
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/7e989bd9/attachment.html>

From mlohry at princeton.edu  Tue May 19 10:32:31 2015
From: mlohry at princeton.edu (Mark Lohry)
Date: Tue, 19 May 2015 11:32:31 -0400
Subject: [petsc-users] DMPlex examples in parallel/3D?
In-Reply-To: <CAMYG4GnKx2D+cc8+ELDA=5Va-Po=XLSkdCZzC9Jm+NdAE0Cg5w@mail.gmail.com>
References: <555B5358.5070501@princeton.edu>
	<CAMYG4GnKx2D+cc8+ELDA=5Va-Po=XLSkdCZzC9Jm+NdAE0Cg5w@mail.gmail.com>
Message-ID: <555B578F.3080809@princeton.edu>

Great thanks, I'll look more into this.

Question on the dofs that wasn't clear from the manual - are the dofs on 
cells, faces, edges, and vertices *all* actual degrees of freedom / 
solution unknowns in the system to solve, or does one say the dofs on 
the cells represent actual solution unknowns, and face dofs are just 
things to be communicated for inter-cell flux computation? e.g. say i 
was doing the simplest cellwise-constant finite volume scheme for a 
scalar; do i have 1 dof per cell, and zero dofs anywhere else?



>
> You can easily get started by just calling DMPlexDistribute() on a 
> serial mesh.
>

perfect, i'll start with that.


>
>
> I do cG FEM in SNES ex12 and ex62.
>


Great thanks, didn't see these linked from the DMPlex doc pages.

From knepley at gmail.com  Tue May 19 10:38:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 May 2015 10:38:10 -0500
Subject: [petsc-users] DMPlex examples in parallel/3D?
In-Reply-To: <555B578F.3080809@princeton.edu>
References: <555B5358.5070501@princeton.edu>
	<CAMYG4GnKx2D+cc8+ELDA=5Va-Po=XLSkdCZzC9Jm+NdAE0Cg5w@mail.gmail.com>
	<555B578F.3080809@princeton.edu>
Message-ID: <CAMYG4GkoGz36qcT9sUrC4qvVm5+XBVhBs6-wi1aB8JUnkbKz3A@mail.gmail.com>

On Tue, May 19, 2015 at 10:32 AM, Mark Lohry <mlohry at princeton.edu> wrote:

> Great thanks, I'll look more into this.
>
> Question on the dofs that wasn't clear from the manual - are the dofs on
> cells, faces, edges, and vertices *all* actual degrees of freedom /
> solution unknowns in the system to solve, or does one say the dofs on the
> cells represent actual solution unknowns, and face dofs are just things to
> be communicated for inter-cell flux computation? e.g. say i was doing the
> simplest cellwise-constant finite volume scheme for a scalar; do i have 1
> dof per cell, and zero dofs anywhere else?
>

There are many ways you could imagine structuring the data. The thing about
PetscSection is that is easily allows
you to pick any one of those.

In a FV example for PETSc, TS ex11, we choose to store only cell data. Then
in loops over the faces, you

  1) Call DMPlexGetSupport(dm, face, &cells) to get the two cells

  2) Call DMPlexGetPointGlobalRead(dm, cell[i], solArray, &val) to get the
cell data


>
>> You can easily get started by just calling DMPlexDistribute() on a serial
>> mesh.
>>
>>
> perfect, i'll start with that.
>
>
>
>>
>> I do cG FEM in SNES ex12 and ex62.
>>
>>
> Great thanks, didn't see these linked from the DMPlex doc pages.
>

They are in the dev docs I believe.

  Thanks,

    Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/246f97bf/attachment-0001.html>

From gbisht at lbl.gov  Tue May 19 10:54:19 2015
From: gbisht at lbl.gov (Gautam Bisht)
Date: Tue, 19 May 2015 08:54:19 -0700
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <CAPz1Tnd-xt5h4QA3m6NvKMNUo6U9+VZfNE3bP72Hq2gyaOu5QA@mail.gmail.com>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
	<alpine.LFD.2.11.1505181418310.18494@asterix>
	<CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>
	<alpine.LFD.2.11.1505181430440.18494@asterix>
	<CAPz1Tnfdpiubs9FJSB7puG5NMNnTFNV43sZh7zDEKr09Okw_jA@mail.gmail.com>
	<CAPz1Tnd-xt5h4QA3m6NvKMNUo6U9+VZfNE3bP72Hq2gyaOu5QA@mail.gmail.com>
Message-ID: <CAPz1Tnc85YbNU4SsoAPGmOUceOA=ftdDNgpE8nV7c3Bm+cnZ=A@mail.gmail.com>

For both, master and v3.5.3+Satish's patch, I get the C preprocessor error
related to installation of sowing. Attached below are log files.

-Gautam.

PS: My previous emails bounced, so I'm re-trying to send log files as tar
file.


On Tue, May 19, 2015 at 8:14 AM, Gautam Bisht <gbisht at lbl.gov> wrote:

> Please disregard the logfile in the last email. The correct ones are
> attached below.
>
> -Gautam.
>
> On Tue, May 19, 2015 at 8:12 AM, Gautam Bisht <gbisht at lbl.gov> wrote:
>
>> For both, master and v3.5.3+Satish's patch, I get the C preprocessor
>> error related to installation of sowing. Attached below are log files.
>>
>> -Gautam.
>>
>> On Mon, May 18, 2015 at 12:33 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>
>>> Can you try using petsc development branch 'master' and see if
>>> it works? It should also add in the c++ compiler to hdf5 configure..
>>>
>>> Wrt petsc-3.5 - perhaps the attached patch will work..
>>>
>>> Satish
>>>
>>> On Mon, 18 May 2015, Gautam Bisht wrote:
>>>
>>> > Attached below is the config.log for HDF5.
>>> >
>>> > -Gautam.
>>> >
>>> > On Mon, May 18, 2015 at 12:18 PM, Satish Balay <balay at mcs.anl.gov>
>>> wrote:
>>> >
>>> > > On Mon, 18 May 2015, Matthew Knepley wrote:
>>> > >
>>> > > > On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov>
>>> wrote:
>>> > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > While trying to install v3.5.3 with intel 15.0.0 on a linux
>>> machine,
>>> > > I'm
>>> > > > > getting an error related to C++ preprocessor. Any help to
>>> resolve the
>>> > > issue
>>> > > > > would be greatly appreciated.
>>> > > > >
>>> > > > > Attached below is the logfile.
>>> > > > >
>>> > > >
>>> > > > The HDF5 configure is failing because it says your preprocessor
>>> does not
>>> > > > work. We would need
>>> > > > to see
>>> > > >
>>> > > >
>>> > >
>>> /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log
>>> > >
>>> > > I think the issue is - hdf5 configure is looking for c++ compiler -
>>> > > but PETSc configure is not providing this info to hdf5 configure.
>>> > >
>>> > > Satish
>>> > >
>>> >
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/dba5bf58/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: petsc-installation-lawrencium.tar.gz
Type: application/x-gzip
Size: 1156200 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/dba5bf58/attachment-0001.bin>

From balay at mcs.anl.gov  Tue May 19 11:13:12 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 19 May 2015 11:13:12 -0500
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <CAPz1Tnc85YbNU4SsoAPGmOUceOA=ftdDNgpE8nV7c3Bm+cnZ=A@mail.gmail.com>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
	<alpine.LFD.2.11.1505181418310.18494@asterix>
	<CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>
	<alpine.LFD.2.11.1505181430440.18494@asterix>
	<CAPz1Tnfdpiubs9FJSB7puG5NMNnTFNV43sZh7zDEKr09Okw_jA@mail.gmail.com>
	<CAPz1Tnd-xt5h4QA3m6NvKMNUo6U9+VZfNE3bP72Hq2gyaOu5QA@mail.gmail.com>
	<CAPz1Tnc85YbNU4SsoAPGmOUceOA=ftdDNgpE8nV7c3Bm+cnZ=A@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505191104480.2724@asterix>

sowing defaults to using gcc/g++ [from env] for the builds.
Looks like the default gcc in your path is broken.

>>>>>>>>
configure:4100: gcc -E -traditional-cpp  conftest.c
In file included from /usr/include/features.h:362,
                 from /usr/include/assert.h:38,
                 from conftest.c:12:
/usr/include/sys/cdefs.h:32: error: #error "You need a ISO C conforming compiler to use the glibc headers"
configure:4107: $? = 1
<<<<<<<<<<<

Do you have access to a different gcc - that doesn't give this error?

Alternatively you can try '--download-sowing-cc=icc --download-sowing-cxx=icpc'
and see if that works.


Satish

On Tue, 19 May 2015, Gautam Bisht wrote:

> For both, master and v3.5.3+Satish's patch, I get the C preprocessor error
> related to installation of sowing. Attached below are log files.
> 
> -Gautam.
> 
> PS: My previous emails bounced, so I'm re-trying to send log files as tar
> file.
> 
> 
> On Tue, May 19, 2015 at 8:14 AM, Gautam Bisht <gbisht at lbl.gov> wrote:
> 
> > Please disregard the logfile in the last email. The correct ones are
> > attached below.
> >
> > -Gautam.
> >
> > On Tue, May 19, 2015 at 8:12 AM, Gautam Bisht <gbisht at lbl.gov> wrote:
> >
> >> For both, master and v3.5.3+Satish's patch, I get the C preprocessor
> >> error related to installation of sowing. Attached below are log files.
> >>
> >> -Gautam.
> >>
> >> On Mon, May 18, 2015 at 12:33 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >>
> >>> Can you try using petsc development branch 'master' and see if
> >>> it works? It should also add in the c++ compiler to hdf5 configure..
> >>>
> >>> Wrt petsc-3.5 - perhaps the attached patch will work..
> >>>
> >>> Satish
> >>>
> >>> On Mon, 18 May 2015, Gautam Bisht wrote:
> >>>
> >>> > Attached below is the config.log for HDF5.
> >>> >
> >>> > -Gautam.
> >>> >
> >>> > On Mon, May 18, 2015 at 12:18 PM, Satish Balay <balay at mcs.anl.gov>
> >>> wrote:
> >>> >
> >>> > > On Mon, 18 May 2015, Matthew Knepley wrote:
> >>> > >
> >>> > > > On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov>
> >>> wrote:
> >>> > > >
> >>> > > > > Hi,
> >>> > > > >
> >>> > > > > While trying to install v3.5.3 with intel 15.0.0 on a linux
> >>> machine,
> >>> > > I'm
> >>> > > > > getting an error related to C++ preprocessor. Any help to
> >>> resolve the
> >>> > > issue
> >>> > > > > would be greatly appreciated.
> >>> > > > >
> >>> > > > > Attached below is the logfile.
> >>> > > > >
> >>> > > >
> >>> > > > The HDF5 configure is failing because it says your preprocessor
> >>> does not
> >>> > > > work. We would need
> >>> > > > to see
> >>> > > >
> >>> > > >
> >>> > >
> >>> /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log
> >>> > >
> >>> > > I think the issue is - hdf5 configure is looking for c++ compiler -
> >>> > > but PETSc configure is not providing this info to hdf5 configure.
> >>> > >
> >>> > > Satish
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
> 


From gbisht at lbl.gov  Tue May 19 11:29:44 2015
From: gbisht at lbl.gov (Gautam Bisht)
Date: Tue, 19 May 2015 09:29:44 -0700
Subject: [petsc-users] Installation error: C++ preprocessor
In-Reply-To: <alpine.LFD.2.11.1505191104480.2724@asterix>
References: <CAPz1TndVPejQaLmnAuwp69Oe5aOUsECfRK2fPjzZw+usWHgazQ@mail.gmail.com>
	<CAMYG4Gk4RjRS545gchMv7P_EJg7+Ps8iTQaNDB8cgpJDgKq7Wg@mail.gmail.com>
	<alpine.LFD.2.11.1505181418310.18494@asterix>
	<CAPz1TncxvEU1=WUrcCdbkn0bzONsaMC++9H4sg0oC8d47OZLKA@mail.gmail.com>
	<alpine.LFD.2.11.1505181430440.18494@asterix>
	<CAPz1Tnfdpiubs9FJSB7puG5NMNnTFNV43sZh7zDEKr09Okw_jA@mail.gmail.com>
	<CAPz1Tnd-xt5h4QA3m6NvKMNUo6U9+VZfNE3bP72Hq2gyaOu5QA@mail.gmail.com>
	<CAPz1Tnc85YbNU4SsoAPGmOUceOA=ftdDNgpE8nV7c3Bm+cnZ=A@mail.gmail.com>
	<alpine.LFD.2.11.1505191104480.2724@asterix>
Message-ID: <CAPz1TndSV0dFBF3krBCpxWX2JDbAvmvgw5WE8zzwobwOHNYNDg@mail.gmail.com>

Thanks, Satish. Specifying '--download-sowing-cc=icc
--download-sowing-cxx=icpc', did the trick.

-Gautam.

On Tue, May 19, 2015 at 9:13 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> sowing defaults to using gcc/g++ [from env] for the builds.
> Looks like the default gcc in your path is broken.
>
> >>>>>>>>
> configure:4100: gcc -E -traditional-cpp  conftest.c
> In file included from /usr/include/features.h:362,
>                  from /usr/include/assert.h:38,
>                  from conftest.c:12:
> /usr/include/sys/cdefs.h:32: error: #error "You need a ISO C conforming
> compiler to use the glibc headers"
> configure:4107: $? = 1
> <<<<<<<<<<<
>
> Do you have access to a different gcc - that doesn't give this error?
>
> Alternatively you can try '--download-sowing-cc=icc
> --download-sowing-cxx=icpc'
> and see if that works.
>
>
> Satish
>
> On Tue, 19 May 2015, Gautam Bisht wrote:
>
> > For both, master and v3.5.3+Satish's patch, I get the C preprocessor
> error
> > related to installation of sowing. Attached below are log files.
> >
> > -Gautam.
> >
> > PS: My previous emails bounced, so I'm re-trying to send log files as tar
> > file.
> >
> >
> > On Tue, May 19, 2015 at 8:14 AM, Gautam Bisht <gbisht at lbl.gov> wrote:
> >
> > > Please disregard the logfile in the last email. The correct ones are
> > > attached below.
> > >
> > > -Gautam.
> > >
> > > On Tue, May 19, 2015 at 8:12 AM, Gautam Bisht <gbisht at lbl.gov> wrote:
> > >
> > >> For both, master and v3.5.3+Satish's patch, I get the C preprocessor
> > >> error related to installation of sowing. Attached below are log files.
> > >>
> > >> -Gautam.
> > >>
> > >> On Mon, May 18, 2015 at 12:33 PM, Satish Balay <balay at mcs.anl.gov>
> wrote:
> > >>
> > >>> Can you try using petsc development branch 'master' and see if
> > >>> it works? It should also add in the c++ compiler to hdf5 configure..
> > >>>
> > >>> Wrt petsc-3.5 - perhaps the attached patch will work..
> > >>>
> > >>> Satish
> > >>>
> > >>> On Mon, 18 May 2015, Gautam Bisht wrote:
> > >>>
> > >>> > Attached below is the config.log for HDF5.
> > >>> >
> > >>> > -Gautam.
> > >>> >
> > >>> > On Mon, May 18, 2015 at 12:18 PM, Satish Balay <balay at mcs.anl.gov>
> > >>> wrote:
> > >>> >
> > >>> > > On Mon, 18 May 2015, Matthew Knepley wrote:
> > >>> > >
> > >>> > > > On Mon, May 18, 2015 at 1:35 PM, Gautam Bisht <gbisht at lbl.gov>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > Hi,
> > >>> > > > >
> > >>> > > > > While trying to install v3.5.3 with intel 15.0.0 on a linux
> > >>> machine,
> > >>> > > I'm
> > >>> > > > > getting an error related to C++ preprocessor. Any help to
> > >>> resolve the
> > >>> > > issue
> > >>> > > > > would be greatly appreciated.
> > >>> > > > >
> > >>> > > > > Attached below is the logfile.
> > >>> > > > >
> > >>> > > >
> > >>> > > > The HDF5 configure is failing because it says your preprocessor
> > >>> does not
> > >>> > > > work. We would need
> > >>> > > > to see
> > >>> > > >
> > >>> > > >
> > >>> > >
> > >>>
> /clusterfs/esd/esd2/gbisht/MODELS/petsc3.5.3/lrc-intel-2015-0-090-fort-debug/externalpackages/hdf5-1.8.10-patch1.1/config.log
> > >>> > >
> > >>> > > I think the issue is - hdf5 configure is looking for c++
> compiler -
> > >>> > > but PETSc configure is not providing this info to hdf5 configure.
> > >>> > >
> > >>> > > Satish
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/327f4ba0/attachment.html>

From bsmith at mcs.anl.gov  Tue May 19 15:59:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 19 May 2015 15:59:48 -0500
Subject: [petsc-users] how to set the newton trust region size
In-Reply-To: <55568347.5040402@princeton.edu>
References: <55568347.5040402@princeton.edu>
Message-ID: <EAD5C1DA-7DB6-40C3-874E-2E3B3DCE8F2E@mcs.anl.gov>


  Yeah, setting the initial trust region from ||f|| is nuts. I have changed the code in master to set it based on ||x|| in the master development branch http://www.mcs.anl.gov/petsc/developers/index.html.  If you use a -snes_tr_delta0 .1 then it will limit the update to 10% as measured in the 2 norm. I am not sure the trust region algorithms make sense using the max norm.  

   So now the initial size should be much more reasonable. Please let us know if this does not help.

  Barry

It will still be valuable for you to non-dimensionalize your variables.




> On May 15, 2015, at 6:37 PM, Mark Lohry <mlohry at princeton.edu> wrote:
> 
> I'm having an issue with SNES where the regular newton line search severely overshoots the solution update, ending up with irrecoverable values (negative densities and temperatures in a fluids code), so it seems newtonTR is a better way to go.
> 
> the manual says to set delta0 using -snes_tr_delta0, where the trust region radius is delta = delta0 * norm2( F ). My F values aren't well scaled and have pretty different values so norm2 of the whole thing isn't particularly meaningful. Ideally I'd like to tell SNES to limit the update to say, 10% of the current value of any given unknown. Is this possible?


From jychang48 at gmail.com  Tue May 19 16:30:41 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 19 May 2015 16:30:41 -0500
Subject: [petsc-users] Vec operations load/stores
Message-ID: <CAP2=TMgUHL11j3FXaRAX7pFjCJQDt3dJOMX41FLsaVBPcp3+hw@mail.gmail.com>

Hi everyone,

To somewhat follow up on a previous mail, I have a few questions regarding
the total bytes transferred for the following Vec operations (assuming Vec
size N and sequential mode):

VecTDot()
VecDot()
VecNorm()
VecScale()
VecSet()
VecAXPY()
VecAYPX()
VecWAXPY
VecPointwiseMult()

1) for the first three operations, I am loading two vectors so that's 2*N*8
bytes transferred. For storing, am I simply storing one scalar? Or am I
individually storing all N components as they are being summed up?

2) for the next five operations where I now have a scalar, am I loading it
only once or is it loaded N times?

3) Do any of the above operations "overlap" or depend on one another? For
instance if my solver invokes VecTDot X times does it also
invoke, say, VecPointwiseMult X times?

This is all theoretically speaking (i.e., assuming I am bypassing the cache
and write-allocate policy).

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/8aeba2d9/attachment.html>

From knepley at gmail.com  Tue May 19 16:43:14 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 May 2015 16:43:14 -0500
Subject: [petsc-users] Vec operations load/stores
In-Reply-To: <CAP2=TMgUHL11j3FXaRAX7pFjCJQDt3dJOMX41FLsaVBPcp3+hw@mail.gmail.com>
References: <CAP2=TMgUHL11j3FXaRAX7pFjCJQDt3dJOMX41FLsaVBPcp3+hw@mail.gmail.com>
Message-ID: <CAMYG4GmXV6aaa157z16L++iKG9+PDfTBBFiPWTH2e5oKRPdOhw@mail.gmail.com>

On Tue, May 19, 2015 at 4:30 PM, Justin Chang <jychang48 at gmail.com> wrote:

> Hi everyone,
>
> To somewhat follow up on a previous mail, I have a few questions regarding
> the total bytes transferred for the following Vec operations (assuming Vec
> size N and sequential mode):
>
> VecTDot()
> VecDot()
> VecNorm()
> VecScale()
> VecSet()
> VecAXPY()
> VecAYPX()
> VecWAXPY
> VecPointwiseMult()
>
> 1) for the first three operations, I am loading two vectors so that's
> 2*N*8 bytes transferred. For storing, am I simply storing one scalar? Or am
> I individually storing all N components as they are being summed up?
>

First 2 operations. No need to count the store of 1 scalar.


> 2) for the next five operations where I now have a scalar, am I loading it
> only once or is it loaded N times?
>

Once, so don't bother counting.


> 3) Do any of the above operations "overlap" or depend on one another? For
> instance if my solver invokes VecTDot X times does it also
> invoke, say, VecPointwiseMult X times?
>

No.

  Thanks,

    Matt


> This is all theoretically speaking (i.e., assuming I am bypassing the
> cache and write-allocate policy).
>
> Thanks,
> Justin
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/838e51c7/attachment.html>

From fande.kong at colorado.edu  Tue May 19 17:33:38 2015
From: fande.kong at colorado.edu (Fande Kong)
Date: Tue, 19 May 2015 16:33:38 -0600
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
Message-ID: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>

Hi all,

I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my
understanding, I think MatGetSubMatrix extracts local rows and columns to
form another parallel matrix with the same communicator, while
MatGetSubMatrices extracts both local and remote entities to construct some
sequential matrices. Is it right?

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/10576a57/attachment.html>

From bsmith at mcs.anl.gov  Tue May 19 17:40:20 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 19 May 2015 17:40:20 -0500
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
In-Reply-To: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
References: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
Message-ID: <0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>


> On May 19, 2015, at 5:33 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> 
> Hi all,
> 
> I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my understanding, I think MatGetSubMatrix extracts local rows and columns to form another parallel matrix with the same communicator, while MatGetSubMatrices extracts both local and remote entities to construct some sequential matrices. Is it right?

  Yes except that MatGetSubMatrix can also use remote rows (and columns) for the part that is on a particular processor. That is it is not restricted to only grabbing local rows/columns in the new parallel matrix.


> 
> Thanks,
> 
> Fande,


From fande.kong at colorado.edu  Tue May 19 17:47:46 2015
From: fande.kong at colorado.edu (Fande Kong)
Date: Tue, 19 May 2015 16:47:46 -0600
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
In-Reply-To: <0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>
References: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
	<0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>
Message-ID: <CAN5Wd-+Fhh0Ox3uSsU_666=8=KdCxk4LBUzOqwXgVmnzaDxdqw@mail.gmail.com>

If I am interested in extracting some remote and local rows and columns but
as another sequential matrix, then I have to use  MatGetSubMatrices? There
are no ways to use MatGetSubMatrix to get a sequential matrix? Because I am
only interested in one sequential matrix per core not multi matrices.

Fande,

On Tue, May 19, 2015 at 4:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 19, 2015, at 5:33 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> >
> > Hi all,
> >
> > I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my
> understanding, I think MatGetSubMatrix extracts local rows and columns to
> form another parallel matrix with the same communicator, while
> MatGetSubMatrices extracts both local and remote entities to construct some
> sequential matrices. Is it right?
>
>   Yes except that MatGetSubMatrix can also use remote rows (and columns)
> for the part that is on a particular processor. That is it is not
> restricted to only grabbing local rows/columns in the new parallel matrix.
>
>
> >
> > Thanks,
> >
> > Fande,
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/5f2a0060/attachment.html>

From bsmith at mcs.anl.gov  Tue May 19 17:54:10 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 19 May 2015 17:54:10 -0500
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
In-Reply-To: <CAN5Wd-+Fhh0Ox3uSsU_666=8=KdCxk4LBUzOqwXgVmnzaDxdqw@mail.gmail.com>
References: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
	<0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>
	<CAN5Wd-+Fhh0Ox3uSsU_666=8=KdCxk4LBUzOqwXgVmnzaDxdqw@mail.gmail.com>
Message-ID: <E6A8E350-F77A-446A-BFA8-804D41F6F239@mcs.anl.gov>


> On May 19, 2015, at 5:47 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> 
> If I am interested in extracting some remote and local rows and columns but as another sequential matrix, then I have to use  MatGetSubMatrices?

  Yes

> There are no ways to use MatGetSubMatrix to get a sequential matrix?

  No

> Because I am only interested in one sequential matrix per core not multi matrices.

  You can just ask for one matrix with MatGetSubMatrices() will give you one sequential matrix per MPI process. 


> 
> Fande,
> 
> On Tue, May 19, 2015 at 4:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On May 19, 2015, at 5:33 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> >
> > Hi all,
> >
> > I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my understanding, I think MatGetSubMatrix extracts local rows and columns to form another parallel matrix with the same communicator, while MatGetSubMatrices extracts both local and remote entities to construct some sequential matrices. Is it right?
> 
>   Yes except that MatGetSubMatrix can also use remote rows (and columns) for the part that is on a particular processor. That is it is not restricted to only grabbing local rows/columns in the new parallel matrix.
> 
> 
> >
> > Thanks,
> >
> > Fande,
> 
> 


From fande.kong at colorado.edu  Tue May 19 17:56:04 2015
From: fande.kong at colorado.edu (Fande Kong)
Date: Tue, 19 May 2015 16:56:04 -0600
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
In-Reply-To: <E6A8E350-F77A-446A-BFA8-804D41F6F239@mcs.anl.gov>
References: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
	<0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>
	<CAN5Wd-+Fhh0Ox3uSsU_666=8=KdCxk4LBUzOqwXgVmnzaDxdqw@mail.gmail.com>
	<E6A8E350-F77A-446A-BFA8-804D41F6F239@mcs.anl.gov>
Message-ID: <CAN5Wd-+BL9R+0Yk7zbAoD0CtBM=Q5NLux1==WtsYaa1F=do45A@mail.gmail.com>

Thanks, Barry. I see.

On Tue, May 19, 2015 at 4:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 19, 2015, at 5:47 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> >
> > If I am interested in extracting some remote and local rows and columns
> but as another sequential matrix, then I have to use  MatGetSubMatrices?
>
>   Yes
>
> > There are no ways to use MatGetSubMatrix to get a sequential matrix?
>
>   No
>
> > Because I am only interested in one sequential matrix per core not multi
> matrices.
>
>   You can just ask for one matrix with MatGetSubMatrices() will give you
> one sequential matrix per MPI process.
>
>
> >
> > Fande,
> >
> > On Tue, May 19, 2015 at 4:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On May 19, 2015, at 5:33 PM, Fande Kong <fande.kong at colorado.edu>
> wrote:
> > >
> > > Hi all,
> > >
> > > I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my
> understanding, I think MatGetSubMatrix extracts local rows and columns to
> form another parallel matrix with the same communicator, while
> MatGetSubMatrices extracts both local and remote entities to construct some
> sequential matrices. Is it right?
> >
> >   Yes except that MatGetSubMatrix can also use remote rows (and columns)
> for the part that is on a particular processor. That is it is not
> restricted to only grabbing local rows/columns in the new parallel matrix.
> >
> >
> > >
> > > Thanks,
> > >
> > > Fande,
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/3b7a9493/attachment-0001.html>

From jychang48 at gmail.com  Tue May 19 18:09:25 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Tue, 19 May 2015 18:09:25 -0500
Subject: [petsc-users] Vec operations load/stores
In-Reply-To: <CAMYG4GmXV6aaa157z16L++iKG9+PDfTBBFiPWTH2e5oKRPdOhw@mail.gmail.com>
References: <CAP2=TMgUHL11j3FXaRAX7pFjCJQDt3dJOMX41FLsaVBPcp3+hw@mail.gmail.com>
	<CAMYG4GmXV6aaa157z16L++iKG9+PDfTBBFiPWTH2e5oKRPdOhw@mail.gmail.com>
Message-ID: <CAP2=TMiN3OTOhUNjCa4fztJoimanGdOkFgFrZXtwxWkemqeSxw@mail.gmail.com>

Okay, thanks

On Tue, May 19, 2015 at 4:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, May 19, 2015 at 4:30 PM, Justin Chang <jychang48 at gmail.com> wrote:
>
>> Hi everyone,
>>
>> To somewhat follow up on a previous mail, I have a few questions
>> regarding the total bytes transferred for the following Vec operations
>> (assuming Vec size N and sequential mode):
>>
>> VecTDot()
>> VecDot()
>> VecNorm()
>> VecScale()
>> VecSet()
>> VecAXPY()
>> VecAYPX()
>> VecWAXPY
>> VecPointwiseMult()
>>
>> 1) for the first three operations, I am loading two vectors so that's
>> 2*N*8 bytes transferred. For storing, am I simply storing one scalar? Or am
>> I individually storing all N components as they are being summed up?
>>
>
> First 2 operations. No need to count the store of 1 scalar.
>
>
>> 2) for the next five operations where I now have a scalar, am I loading
>> it only once or is it loaded N times?
>>
>
> Once, so don't bother counting.
>
>
>> 3) Do any of the above operations "overlap" or depend on one another? For
>> instance if my solver invokes VecTDot X times does it also
>> invoke, say, VecPointwiseMult X times?
>>
>
> No.
>
>   Thanks,
>
>     Matt
>
>
>> This is all theoretically speaking (i.e., assuming I am bypassing the
>> cache and write-allocate policy).
>>
>> Thanks,
>> Justin
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/603886f7/attachment.html>

From knepley at gmail.com  Tue May 19 18:36:49 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 19 May 2015 18:36:49 -0500
Subject: [petsc-users] can MatGetSubMatrix extract remote rows?
In-Reply-To: <E6A8E350-F77A-446A-BFA8-804D41F6F239@mcs.anl.gov>
References: <CAN5Wd-K8aRZ3ZSFwkesUaO2BDANftpsXSr-oSq4r8az5FP9h9g@mail.gmail.com>
	<0D1A5DB2-DDBF-4765-A488-E077AB316579@mcs.anl.gov>
	<CAN5Wd-+Fhh0Ox3uSsU_666=8=KdCxk4LBUzOqwXgVmnzaDxdqw@mail.gmail.com>
	<E6A8E350-F77A-446A-BFA8-804D41F6F239@mcs.anl.gov>
Message-ID: <CAMYG4GkXm38Rm4ng5RqYQqwisUpmqrNd0-QHdgQgepd9bCg1ig@mail.gmail.com>

On Tue, May 19, 2015 at 5:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 19, 2015, at 5:47 PM, Fande Kong <fande.kong at colorado.edu> wrote:
> >
> > If I am interested in extracting some remote and local rows and columns
> but as another sequential matrix, then I have to use  MatGetSubMatrices?
>
>   Yes
>
> > There are no ways to use MatGetSubMatrix to get a sequential matrix?
>
>   No


Well, you could use MatGetSubMatrix() where only 1 proc gets rows and
columns, then extract the
serial submatrix.

   Matt


>
> > Because I am only interested in one sequential matrix per core not multi
> matrices.
>
>   You can just ask for one matrix with MatGetSubMatrices() will give you
> one sequential matrix per MPI process.
>
>
> >
> > Fande,
> >
> > On Tue, May 19, 2015 at 4:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On May 19, 2015, at 5:33 PM, Fande Kong <fande.kong at colorado.edu>
> wrote:
> > >
> > > Hi all,
> > >
> > > I am confused with MatGetSubMatrix and MatGetSubMatrices.  In my
> understanding, I think MatGetSubMatrix extracts local rows and columns to
> form another parallel matrix with the same communicator, while
> MatGetSubMatrices extracts both local and remote entities to construct some
> sequential matrices. Is it right?
> >
> >   Yes except that MatGetSubMatrix can also use remote rows (and columns)
> for the part that is on a particular processor. That is it is not
> restricted to only grabbing local rows/columns in the new parallel matrix.
> >
> >
> > >
> > > Thanks,
> > >
> > > Fande,
> >
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150519/291144d4/attachment.html>

From rongliang.chan at gmail.com  Wed May 20 22:27:21 2015
From: rongliang.chan at gmail.com (Rongliang Chen)
Date: Thu, 21 May 2015 11:27:21 +0800
Subject: [petsc-users] About the FV function
Message-ID: <CAOoA1QUDkPf0Hz5wDVOWWuuZ8pUceok6pyx5avhn9SMLxzNnkQ@mail.gmail.com>

Hi there,

I have a question about the finite volume formulation. In function
TSComputeRHSFunction_DMPlex, when we compute the flux (line 617), why we do
not multiply the face area? In the formulation of the face flux, I think we
need to multiply the face area and divide the element volume.

Best,
Rongliang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150521/9761e024/attachment.html>

From knepley at gmail.com  Thu May 21 07:13:32 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 21 May 2015 07:13:32 -0500
Subject: [petsc-users] About the FV function
In-Reply-To: <CAOoA1QUDkPf0Hz5wDVOWWuuZ8pUceok6pyx5avhn9SMLxzNnkQ@mail.gmail.com>
References: <CAOoA1QUDkPf0Hz5wDVOWWuuZ8pUceok6pyx5avhn9SMLxzNnkQ@mail.gmail.com>
Message-ID: <CAMYG4Gm6DvwSZaj4LWy61QKmJrLu-TwFipevuCk+q_S3-+u4PQ@mail.gmail.com>

On Wed, May 20, 2015 at 10:27 PM, Rongliang Chen <rongliang.chan at gmail.com>
wrote:

> Hi there,
>
> I have a question about the finite volume formulation. In function
> TSComputeRHSFunction_DMPlex, when we compute the flux (line 617), why we do
> not multiply the face area? In the formulation of the face flux, I think we
> need to multiply the face area and divide the element volume. '
>

The normals are area weighted. I will indicate this in the documentation:

https://bitbucket.org/petsc/petsc/src/5a737092961693f09ed5214c2133c4a7b76f72d7/src/dm/impls/plex/plexgeometry.c?at=master#cl-1208

  Thanks,

     Matt

Best,
> Rongliang
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150521/bc6379ee/attachment.html>

From Eric.Chamberland at giref.ulaval.ca  Fri May 22 09:05:00 2015
From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland)
Date: Fri, 22 May 2015 10:05:00 -0400
Subject: [petsc-users] Block size
In-Reply-To: <C64D0FBC-3D56-44AB-A6FD-7E8ACDD3BE77@mcs.anl.gov>
References: <CAFdHYfuNerLEAJQZ44XAjCWRRgwOm-nEtAsbSA+VTOCdpJ_AUA@mail.gmail.com>
	<C64D0FBC-3D56-44AB-A6FD-7E8ACDD3BE77@mcs.anl.gov>
Message-ID: <555F378C.5000706@giref.ulaval.ca>

Hi,

I have another question about this.

I am actually working on a multi-variable problem (chemical species and 
voltage in brain), but all variables are not fully coupled with others. 
  Let's say I have scalar variables A, B, C and D coupled like this:

   A B C D
A x     x
B   x   x
C     x x
D x x x x

I first number my variables continuously on each vertex, so that I have 
a "block size" of 4 on each vertex.

With an {SEQ,MPI}AIJ matrix, I can preallocate the matrix by "scalar 
lines" them minimizing the memory allocated for the structure.

My question is:  Do all preconditionners are able to handle "sparse 
blocks" when block_size > 1 in {SEQ,MPI}AIJ matrix?  Do they still have 
some gain with "sparse blocks"?

Right now, I am over-allocation {SEQ,MPI}AIJ matrix since I use 
MatXAIJSetPreallocation with bs=4...  But I am thinking of writing a 
different function to not over-allocate memory when a block_size>1 
{SEQ,MPI}AIJ matrix with "sparse blocks" is used.  For example, for a 2d 
problem, if I use bs=4 I have:

   Mat Object:  (Solveur_Direct)   1 MPI processes
     type: seqaij
     rows=8588, cols=8588, bs=4
     total: nonzeros=232816, allocated nonzeros=234832
     total number of mallocs used during MatSetValues calls =0
       using I-node routines: found 2147 nodes, limit used is 5

but if I use bs=1 I have:

   Mat Object:  (Solveur_Direct)   1 MPI processes
     type: seqaij
     rows=8588, cols=8588
     total: nonzeros=100516, allocated nonzeros=100516
     total number of mallocs used during MatSetValues calls =0
       not using I-node routines

which is 2 times less memory for this (toy)-problem.  I am partly 
responsible for this, since I count non-zeros for blocks because I am 
forced to do so if use MatXAIJSetPreallocation.  Since we think to go 
very large, let's say 100M unknowns, I want to avoid such 
over-allocation.  But before modifying this, I am querying your knowledge...

thanks!

Eric


On 10/09/2014 04:19 AM, Barry Smith wrote:
>
> On Oct 9, 2014, at 3:12 AM, Natacha BEREUX <natacha.bereux at gmail.com> wrote:
>
>> Dear PETSc users,
>>
>> I am a bit confused  about blocksizes in MPI matrices.
>>
>> I define a MPI matrix A with a certain blocksize, let say bs = 3.
>> As far as I understand, the sparse pattern of the matrix A is made of square blocks of size 3x3, regardless of the actual values of the terms of  A.
>> Am I right ?
>
>     Only if you use the BAIJ or SBAIJ matrix format. If you use AIJ then it still stores the values without regard to the block size but it carries knowledge of the block size around and that gets used in some places such as the GAMG solver.
>>
>> The matrix A is distributed among several processors: does the local number of rows of A on each processor have to be a multiple of bs   ?
>
>    Correct
>>
>> I want to use a multigrid preconditionner to  solve the linear system of matrix A : A x = b
>> Is it mandatory to define the right hand side b as a vector with the same blocksize bs=3 ?
>
>    It probably doesn?t matter, but if you know something has an associated block size it is generally best to capture it as soon as you can in the problem.
>
>     Barry
>
>>
>> Thank you very much of your help,
>> Best regards,
>> Natacha


From daaugusto at gmail.com  Fri May 22 09:50:01 2015
From: daaugusto at gmail.com (Douglas A. Augusto)
Date: Fri, 22 May 2015 11:50:01 -0300
Subject: [petsc-users] Current state of OpenMP support in PETSc
Message-ID: <20150522145001.GA1163@localhost.localdomain>

Dear PETSc developers:

After trying for a while to get PETSc to run using threads (in particular
OpenMP), I realized that the threading support seems to have been recently
removed from PETSc, both OpenMP and Pthread (commit 163f1a7).

So, will the threading support be eventually reintroduced in PETSc? What are
the plans regarding OpenMP?

Thanks in advance.


Best regards,

-- 
Douglas A. Augusto

From bsmith at mcs.anl.gov  Fri May 22 10:36:36 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 22 May 2015 10:36:36 -0500
Subject: [petsc-users] Block size
In-Reply-To: <555F378C.5000706@giref.ulaval.ca>
References: <CAFdHYfuNerLEAJQZ44XAjCWRRgwOm-nEtAsbSA+VTOCdpJ_AUA@mail.gmail.com>
	<C64D0FBC-3D56-44AB-A6FD-7E8ACDD3BE77@mcs.anl.gov>
	<555F378C.5000706@giref.ulaval.ca>
Message-ID: <84DEA2E0-BAEB-48D2-A5C1-CD1B96140005@mcs.anl.gov>


   Eric,

     There is a performance tradeoff between keeping the blocks full (with some zero padding)(which also allows one to use the BAIJ format) and treating them as fully sparse. Where one should switch between the formats is problem dependent and I don't even have any good rule of thumb for knowing when to switch.  My recommendation is to write the preallocation code general enough to allow trivially switching between the two representations; that is, have code that preallocates by block and code that preallocates by point and have a run time switch between them. With that capability you will always have the freedom to use which every approach makes sense for a given run; for example for smaller runs you likely just want to run for speed and use the blocked preallocation, to squeeze the largest problem you can run you would use the point format and it will be somewhat slower but at least you will not run out of memory.

   Barry



> On May 22, 2015, at 9:05 AM, Eric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
> 
> Hi,
> 
> I have another question about this.
> 
> I am actually working on a multi-variable problem (chemical species and voltage in brain), but all variables are not fully coupled with others.  Let's say I have scalar variables A, B, C and D coupled like this:
> 
>  A B C D
> A x     x
> B   x   x
> C     x x
> D x x x x
> 
> I first number my variables continuously on each vertex, so that I have a "block size" of 4 on each vertex.
> 
> With an {SEQ,MPI}AIJ matrix, I can preallocate the matrix by "scalar lines" them minimizing the memory allocated for the structure.
> 
> My question is:  Do all preconditionners are able to handle "sparse blocks" when block_size > 1 in {SEQ,MPI}AIJ matrix?  Do they still have some gain with "sparse blocks"?
> 
> Right now, I am over-allocation {SEQ,MPI}AIJ matrix since I use MatXAIJSetPreallocation with bs=4...  But I am thinking of writing a different function to not over-allocate memory when a block_size>1 {SEQ,MPI}AIJ matrix with "sparse blocks" is used.  For example, for a 2d problem, if I use bs=4 I have:
> 
>  Mat Object:  (Solveur_Direct)   1 MPI processes
>    type: seqaij
>    rows=8588, cols=8588, bs=4
>    total: nonzeros=232816, allocated nonzeros=234832
>    total number of mallocs used during MatSetValues calls =0
>      using I-node routines: found 2147 nodes, limit used is 5
> 
> but if I use bs=1 I have:
> 
>  Mat Object:  (Solveur_Direct)   1 MPI processes
>    type: seqaij
>    rows=8588, cols=8588
>    total: nonzeros=100516, allocated nonzeros=100516
>    total number of mallocs used during MatSetValues calls =0
>      not using I-node routines
> 
> which is 2 times less memory for this (toy)-problem.  I am partly responsible for this, since I count non-zeros for blocks because I am forced to do so if use MatXAIJSetPreallocation.  Since we think to go very large, let's say 100M unknowns, I want to avoid such over-allocation.  But before modifying this, I am querying your knowledge...
> 
> thanks!
> 
> Eric
> 
> 
> On 10/09/2014 04:19 AM, Barry Smith wrote:
>> 
>> On Oct 9, 2014, at 3:12 AM, Natacha BEREUX <natacha.bereux at gmail.com> wrote:
>> 
>>> Dear PETSc users,
>>> 
>>> I am a bit confused  about blocksizes in MPI matrices.
>>> 
>>> I define a MPI matrix A with a certain blocksize, let say bs = 3.
>>> As far as I understand, the sparse pattern of the matrix A is made of square blocks of size 3x3, regardless of the actual values of the terms of  A.
>>> Am I right ?
>> 
>>    Only if you use the BAIJ or SBAIJ matrix format. If you use AIJ then it still stores the values without regard to the block size but it carries knowledge of the block size around and that gets used in some places such as the GAMG solver.
>>> 
>>> The matrix A is distributed among several processors: does the local number of rows of A on each processor have to be a multiple of bs   ?
>> 
>>   Correct
>>> 
>>> I want to use a multigrid preconditionner to  solve the linear system of matrix A : A x = b
>>> Is it mandatory to define the right hand side b as a vector with the same blocksize bs=3 ?
>> 
>>   It probably doesn?t matter, but if you know something has an associated block size it is generally best to capture it as soon as you can in the problem.
>> 
>>    Barry
>> 
>>> 
>>> Thank you very much of your help,
>>> Best regards,
>>> Natacha
> 


From Eric.Chamberland at giref.ulaval.ca  Fri May 22 11:14:41 2015
From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland)
Date: Fri, 22 May 2015 12:14:41 -0400
Subject: [petsc-users] Block size
In-Reply-To: <84DEA2E0-BAEB-48D2-A5C1-CD1B96140005@mcs.anl.gov>
References: <CAFdHYfuNerLEAJQZ44XAjCWRRgwOm-nEtAsbSA+VTOCdpJ_AUA@mail.gmail.com>
	<C64D0FBC-3D56-44AB-A6FD-7E8ACDD3BE77@mcs.anl.gov>
	<555F378C.5000706@giref.ulaval.ca>
	<84DEA2E0-BAEB-48D2-A5C1-CD1B96140005@mcs.anl.gov>
Message-ID: <555F55F1.9050006@giref.ulaval.ca>

On 05/22/2015 11:36 AM, Barry Smith wrote:
>
>     Eric,
>
 > to squeeze the largest problem you can run you would use the point 
format and it will be somewhat slower but at least you will not run out 
of memory.

Ok, my algorithm is already general and can do both.  But my main worry is:

Do all preconditionners are able to handle "sparse blocks" when 
block_size > 1 in {SEQ,MPI}AIJ matrix?  Do they still have some gain 
with "sparse blocks"?  (for example, algebraic multi-grid will be able 
to pick lines "per-block" when block_size > 1 but blocks are "sparses")?

Maybe the answer is trivial, but we were discussing it here and we got 
different positions on this..

Thanks,

Eric


From bsmith at mcs.anl.gov  Fri May 22 11:26:51 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 22 May 2015 11:26:51 -0500
Subject: [petsc-users] Block size
In-Reply-To: <555F55F1.9050006@giref.ulaval.ca>
References: <CAFdHYfuNerLEAJQZ44XAjCWRRgwOm-nEtAsbSA+VTOCdpJ_AUA@mail.gmail.com>
	<C64D0FBC-3D56-44AB-A6FD-7E8ACDD3BE77@mcs.anl.gov>
	<555F378C.5000706@giref.ulaval.ca>
	<84DEA2E0-BAEB-48D2-A5C1-CD1B96140005@mcs.anl.gov>
	<555F55F1.9050006@giref.ulaval.ca>
Message-ID: <87F54FAD-56C0-48D7-8A21-86DE42AE10C3@mcs.anl.gov>


> On May 22, 2015, at 11:14 AM, Eric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
> 
> On 05/22/2015 11:36 AM, Barry Smith wrote:
>> 
>>    Eric,
>> 
> > to squeeze the largest problem you can run you would use the point format and it will be somewhat slower but at least you will not run out of memory.
> 
> Ok, my algorithm is already general and can do both.  But my main worry is:
> 
> Do all preconditionners are able to handle "sparse blocks" when block_size > 1 in {SEQ,MPI}AIJ matrix?  Do they still have some gain with "sparse blocks"?  

   This depends on the particular algorithm and the particular implementation. There is no general statement. Because of the various contending issues in terms of performance you can only try them and see what happens performance wise.

> (for example, algebraic multi-grid will be able to pick lines "per-block" when block_size > 1 but blocks are "sparses")?

   Yes, the GAMG (hypre BoomerAMG) does exactly that. 
> 
> Maybe the answer is trivial, but we were discussing it here and we got different positions on this..
> 
> Thanks,
> 
> Eric
> 


From jed at jedbrown.org  Fri May 22 14:20:14 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 22 May 2015 13:20:14 -0600
Subject: [petsc-users] Current state of OpenMP support in PETSc
In-Reply-To: <20150522145001.GA1163@localhost.localdomain>
References: <20150522145001.GA1163@localhost.localdomain>
Message-ID: <87pp5sv2q9.fsf@jedbrown.org>

"Douglas A. Augusto" <daaugusto at gmail.com> writes:
> So, will the threading support be eventually reintroduced in PETSc? What are
> the plans regarding OpenMP?

The plan is to use conventional OpenMP because that seems to be what
people want and it's easier to maintain than a more radical approach.
We're also developing shared memory MPI-based approaches and would
recommend that you plan to not use OpenMP (it's a horrible programming
model -- oversynchronizing and without a concept of memory locality).
Note that there is huge selection bias -- people almost never publish
when flat MPI runs faster (which it often does for quality
implementations), but are happy to declare OpenMP a success when it
looks favorable.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150522/7a268173/attachment.pgp>

From rtm at utk.edu  Fri May 22 15:02:23 2015
From: rtm at utk.edu (Richard Mills)
Date: Fri, 22 May 2015 13:02:23 -0700
Subject: [petsc-users] Current state of OpenMP support in PETSc
In-Reply-To: <87pp5sv2q9.fsf@jedbrown.org>
References: <20150522145001.GA1163@localhost.localdomain>
	<87pp5sv2q9.fsf@jedbrown.org>
Message-ID: <CAAZcC2cL14ypF+upaFBC=j2KME+SWn3xAxE7mFtFtGa+r4+ufQ@mail.gmail.com>

Jed (and others),

Are there any more concrete plans for MPI 3 shared memory-based approaches
that have developed since some discussions about this topic a few months
ago on PETSc-dev?  Most of the discussion trended towards making better use
of local memories by the use of neighborhood collectives (which sounds
great, and I was intending to look into doing a neighborhood collective
implementation for PetscSF, though I got buried under other things).  We
probably need to also think about doing things like allowing flexibility in
what ghost points get allocated for local vectors obtained from a DM if
neighbors live in the same shared memory domain, etc.  Though this
sounds... messy.

--Richard

On Fri, May 22, 2015 at 12:20 PM, Jed Brown <jed at jedbrown.org> wrote:

> "Douglas A. Augusto" <daaugusto at gmail.com> writes:
> > So, will the threading support be eventually reintroduced in PETSc? What
> are
> > the plans regarding OpenMP?
>
> The plan is to use conventional OpenMP because that seems to be what
> people want and it's easier to maintain than a more radical approach.
> We're also developing shared memory MPI-based approaches and would
> recommend that you plan to not use OpenMP (it's a horrible programming
> model -- oversynchronizing and without a concept of memory locality).
> Note that there is huge selection bias -- people almost never publish
> when flat MPI runs faster (which it often does for quality
> implementations), but are happy to declare OpenMP a success when it
> looks favorable.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150522/02987ab0/attachment.html>

From venkateshgk.j at gmail.com  Sat May 23 02:39:50 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 23 May 2015 13:09:50 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
Message-ID: <CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>

Hi again,

I have installed the Petsc and Slepc in Cray with intel compilers with
Mumps.

I am getting this error when I solve eigenvalue problem with large
matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
factorization phase: Cannot allocate required memory 9632 megabytes

Also it is again not scaling well for small matrices.

Kindly let me know what to do.

cheers,

Venkatesh


On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi,
>>
>> I have attached the log of the command which I gave in the master node:
>> make streams NPMAX=32
>>
>> I dont know why it says 'It appears you have only 1 node'. But other
>> codes run in parallel with good scaling on 8 nodes.
>>
>
> If you look at the STREAMS numbers, you can see that your system is only
> able to support about 2 cores with the
> available memory bandwidth. Thus for bandwidth constrained operations
> (almost everything in sparse linear algebra
> and solvers), your speedup will not be bigger than 2.
>
> Other codes may do well on this machine, but they would be compute
> constrained, using things like DGEMM.
>
>   Thanks,
>
>      Matt
>
>
>> Kindly let me know.
>>
>> Venkatesh
>>
>>
>>
>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>    Run the streams benchmark on this system and send the results.
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>
>>>
>>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>> >
>>> > Hi,
>>> > I have emailed the mumps-user list.
>>> > Actually the cluster has 8 nodes with 16 cores, and other codes scale
>>> well.
>>> > I wanted to ask if this job takes much time, then if I submit on more
>>> cores, I have to increase the icntl(14).. which would again take long time.
>>> >
>>> > So is there another way ?
>>> >
>>> > cheers,
>>> > Venkatesh
>>> >
>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>> > Hi I have attached the performance logs for 2 jobs on different
>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>> cores since it is failing with small value of icntl(14).
>>> >
>>> > 1. performance_log1.txt is run on 8 cores (option given
>>> -mat_mumps_icntl_14 200)
>>> > 2. performance_log2.txt is run on 2 cores (option given
>>> -mat_mumps_icntl_14 85  )
>>> >
>>> > 1) Your number of iterates increased from 7600 to 9600, but that is a
>>> relatively small effect
>>> >
>>> > 2) MUMPS is just taking a lot longer to do forward/backward solve. You
>>> might try emailing
>>> > the list for them. However, I would bet that your system has enough
>>> bandwidth for 2 procs
>>> > and not much more.
>>> >
>>> >   Thanks,
>>> >
>>> >      Matt
>>> >
>>> > Venkatesh
>>> >
>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>> > Hi, Thanks for the information. I now increased the workspace by
>>> adding '-mat_mumps_icntl_14 100'
>>> >
>>> > It works. However, the problem is, if I submit in 1 core I get the
>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>> 3500secs.
>>> >
>>> > Send the output of -log_summary for all performance queries. Otherwise
>>> we are just guessing.
>>> >
>>> >     Matt
>>> >
>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>> >
>>> > Kindly let me know.
>>> >
>>> > Venkatesh
>>> >
>>> >
>>> >
>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>> david.knezevic at akselos.com> wrote:
>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>> > Hi,
>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>> >
>>> > A and B are of sizes 3600x3600
>>> >
>>> > I run with this command :
>>> >
>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>> -st_pc_factor_mat_solver_package mumps'
>>> >
>>> > I get this error: (I get result only when I give 1 or 2 processors)
>>> > Reading COMPLEX matrices from binary files...
>>> > [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> > [0]PETSC ERROR: Error in external library!
>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: INFO(1)=-9, INFO(2)=2024
>>> >
>>> >
>>> > The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>>> In this case you have INFO(1)=-9, which is explained in the manual as:
>>> >
>>> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>> positive, then the number of entries that are missing in S at the moment
>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>> ICNTL(23) should be increased."
>>> >
>>> > This says that you should use ICTNL(14) to increase the working space
>>> size:
>>> >
>>> > "ICNTL(14) is accessed by the host both during the analysis and the
>>> factorization phases. It corresponds to the percentage increase in the
>>> estimated working space. When significant extra fill-in is caused by
>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>> the default value is 20 (which corresponds to a 20 % increase)."
>>> >
>>> > So, for example, you can avoid this error via the following command
>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>> we allow a 30% increase in the workspace instead of the default 20%.
>>> >
>>> > David
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/7829ef05/attachment.html>

From knepley at gmail.com  Sat May 23 06:28:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 23 May 2015 06:28:58 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
Message-ID: <CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>

On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi again,
>
> I have installed the Petsc and Slepc in Cray with intel compilers with
> Mumps.
>
> I am getting this error when I solve eigenvalue problem with large
> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
> factorization phase: Cannot allocate required memory 9632 megabytes
>

It ran out of memory on the node.


> Also it is again not scaling well for small matrices.
>

MUMPS strong scaling for small matrices is not very good. Weak scaling is
looking at big matrices.

  Thanks,

     Matt


> Kindly let me know what to do.
>
> cheers,
>
> Venkatesh
>
>
> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have attached the log of the command which I gave in the master node:
>>> make streams NPMAX=32
>>>
>>> I dont know why it says 'It appears you have only 1 node'. But other
>>> codes run in parallel with good scaling on 8 nodes.
>>>
>>
>> If you look at the STREAMS numbers, you can see that your system is only
>> able to support about 2 cores with the
>> available memory bandwidth. Thus for bandwidth constrained operations
>> (almost everything in sparse linear algebra
>> and solvers), your speedup will not be bigger than 2.
>>
>> Other codes may do well on this machine, but they would be compute
>> constrained, using things like DGEMM.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Kindly let me know.
>>>
>>> Venkatesh
>>>
>>>
>>>
>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>>
>>>>
>>>>    Run the streams benchmark on this system and send the results.
>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>
>>>>
>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi,
>>>> > I have emailed the mumps-user list.
>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes scale
>>>> well.
>>>> > I wanted to ask if this job takes much time, then if I submit on more
>>>> cores, I have to increase the icntl(14).. which would again take long time.
>>>> >
>>>> > So is there another way ?
>>>> >
>>>> > cheers,
>>>> > Venkatesh
>>>> >
>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>> cores since it is failing with small value of icntl(14).
>>>> >
>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>> -mat_mumps_icntl_14 200)
>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>> -mat_mumps_icntl_14 85  )
>>>> >
>>>> > 1) Your number of iterates increased from 7600 to 9600, but that is a
>>>> relatively small effect
>>>> >
>>>> > 2) MUMPS is just taking a lot longer to do forward/backward solve.
>>>> You might try emailing
>>>> > the list for them. However, I would bet that your system has enough
>>>> bandwidth for 2 procs
>>>> > and not much more.
>>>> >
>>>> >   Thanks,
>>>> >
>>>> >      Matt
>>>> >
>>>> > Venkatesh
>>>> >
>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>> adding '-mat_mumps_icntl_14 100'
>>>> >
>>>> > It works. However, the problem is, if I submit in 1 core I get the
>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>> 3500secs.
>>>> >
>>>> > Send the output of -log_summary for all performance queries.
>>>> Otherwise we are just guessing.
>>>> >
>>>> >     Matt
>>>> >
>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>> >
>>>> > Kindly let me know.
>>>> >
>>>> > Venkatesh
>>>> >
>>>> >
>>>> >
>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>> david.knezevic at akselos.com> wrote:
>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>> > Hi,
>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>> >
>>>> > A and B are of sizes 3600x3600
>>>> >
>>>> > I run with this command :
>>>> >
>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>> -st_pc_factor_mat_solver_package mumps'
>>>> >
>>>> > I get this error: (I get result only when I give 1 or 2 processors)
>>>> > Reading COMPLEX matrices from binary files...
>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>> ------------------------------------
>>>> > [0]PETSC ERROR: Error in external library!
>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>> >
>>>> >
>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS manual.
>>>> In this case you have INFO(1)=-9, which is explained in the manual as:
>>>> >
>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>> positive, then the number of entries that are missing in S at the moment
>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>> ICNTL(23) should be increased."
>>>> >
>>>> > This says that you should use ICTNL(14) to increase the working space
>>>> size:
>>>> >
>>>> > "ICNTL(14) is accessed by the host both during the analysis and the
>>>> factorization phases. It corresponds to the percentage increase in the
>>>> estimated working space. When significant extra fill-in is caused by
>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>> >
>>>> > So, for example, you can avoid this error via the following command
>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>> >
>>>> > David
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > -- Norbert Wiener
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > -- Norbert Wiener
>>>> >
>>>>
>>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/9e5e3be0/attachment-0001.html>

From venkateshgk.j at gmail.com  Sat May 23 06:44:21 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 23 May 2015 17:14:21 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
Message-ID: <CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>

Hi,
The same eigenproblem runs with 120 GB RAM in a serial machine in Matlab.

In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right ?

And for small matrices it is having negative scaling i.e 24 core is running
faster.

I have attached the submission script.

Pls see.. Kindly let me know

cheers,
Venkatesh


On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi again,
>>
>> I have installed the Petsc and Slepc in Cray with intel compilers with
>> Mumps.
>>
>> I am getting this error when I solve eigenvalue problem with large
>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>> factorization phase: Cannot allocate required memory 9632 megabytes
>>
>
> It ran out of memory on the node.
>
>
>> Also it is again not scaling well for small matrices.
>>
>
> MUMPS strong scaling for small matrices is not very good. Weak scaling is
> looking at big matrices.
>
>   Thanks,
>
>      Matt
>
>
>> Kindly let me know what to do.
>>
>> cheers,
>>
>> Venkatesh
>>
>>
>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have attached the log of the command which I gave in the master node:
>>>> make streams NPMAX=32
>>>>
>>>> I dont know why it says 'It appears you have only 1 node'. But other
>>>> codes run in parallel with good scaling on 8 nodes.
>>>>
>>>
>>> If you look at the STREAMS numbers, you can see that your system is only
>>> able to support about 2 cores with the
>>> available memory bandwidth. Thus for bandwidth constrained operations
>>> (almost everything in sparse linear algebra
>>> and solvers), your speedup will not be bigger than 2.
>>>
>>> Other codes may do well on this machine, but they would be compute
>>> constrained, using things like DGEMM.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Kindly let me know.
>>>>
>>>> Venkatesh
>>>>
>>>>
>>>>
>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>>
>>>>>    Run the streams benchmark on this system and send the results.
>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>
>>>>>
>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi,
>>>>> > I have emailed the mumps-user list.
>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>> scale well.
>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>> time.
>>>>> >
>>>>> > So is there another way ?
>>>>> >
>>>>> > cheers,
>>>>> > Venkatesh
>>>>> >
>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>> venkateshgk.j at gmail.com> wrote:
>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>> cores since it is failing with small value of icntl(14).
>>>>> >
>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>> -mat_mumps_icntl_14 200)
>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>> -mat_mumps_icntl_14 85  )
>>>>> >
>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that is
>>>>> a relatively small effect
>>>>> >
>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward solve.
>>>>> You might try emailing
>>>>> > the list for them. However, I would bet that your system has enough
>>>>> bandwidth for 2 procs
>>>>> > and not much more.
>>>>> >
>>>>> >   Thanks,
>>>>> >
>>>>> >      Matt
>>>>> >
>>>>> > Venkatesh
>>>>> >
>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>> venkateshgk.j at gmail.com> wrote:
>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>> adding '-mat_mumps_icntl_14 100'
>>>>> >
>>>>> > It works. However, the problem is, if I submit in 1 core I get the
>>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>>> 3500secs.
>>>>> >
>>>>> > Send the output of -log_summary for all performance queries.
>>>>> Otherwise we are just guessing.
>>>>> >
>>>>> >     Matt
>>>>> >
>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>> >
>>>>> > Kindly let me know.
>>>>> >
>>>>> > Venkatesh
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>> david.knezevic at akselos.com> wrote:
>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>> venkateshgk.j at gmail.com> wrote:
>>>>> > Hi,
>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>> >
>>>>> > A and B are of sizes 3600x3600
>>>>> >
>>>>> > I run with this command :
>>>>> >
>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>> >
>>>>> > I get this error: (I get result only when I give 1 or 2 processors)
>>>>> > Reading COMPLEX matrices from binary files...
>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>> > [0]PETSC ERROR: Error in external library!
>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>> >
>>>>> >
>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>> as:
>>>>> >
>>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>>> positive, then the number of entries that are missing in S at the moment
>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>> ICNTL(23) should be increased."
>>>>> >
>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>> space size:
>>>>> >
>>>>> > "ICNTL(14) is accessed by the host both during the analysis and the
>>>>> factorization phases. It corresponds to the percentage increase in the
>>>>> estimated working space. When significant extra fill-in is caused by
>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>> >
>>>>> > So, for example, you can avoid this error via the following command
>>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>>> >
>>>>> > David
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > -- Norbert Wiener
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > -- Norbert Wiener
>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/6c8c2f04/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: script
Type: application/octet-stream
Size: 376 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/6c8c2f04/attachment.obj>

From knepley at gmail.com  Sat May 23 06:47:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 23 May 2015 06:47:39 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
Message-ID: <CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>

On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
> The same eigenproblem runs with 120 GB RAM in a serial machine in Matlab.
>
> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right ?
>

I do not know how MUMPS allocates memory, but the message is unambiguous.
Also,
this is concerned with the memory available per node. Do you know how many
processes
per node were scheduled? The message below indicates that it was trying to
allocate 10G
for one process.


> And for small matrices it is having negative scaling i.e 24 core is
> running faster.
>

Yes, for strong scaling you always get slowdown eventually since overheads
dominate
work, see Amdahl's Law.

  Thanks,

     Matt


> I have attached the submission script.
>
> Pls see.. Kindly let me know
>
> cheers,
> Venkatesh
>
>
> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi again,
>>>
>>> I have installed the Petsc and Slepc in Cray with intel compilers with
>>> Mumps.
>>>
>>> I am getting this error when I solve eigenvalue problem with large
>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>
>>
>> It ran out of memory on the node.
>>
>>
>>> Also it is again not scaling well for small matrices.
>>>
>>
>> MUMPS strong scaling for small matrices is not very good. Weak scaling is
>> looking at big matrices.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Kindly let me know what to do.
>>>
>>> cheers,
>>>
>>> Venkatesh
>>>
>>>
>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have attached the log of the command which I gave in the master
>>>>> node: make streams NPMAX=32
>>>>>
>>>>> I dont know why it says 'It appears you have only 1 node'. But other
>>>>> codes run in parallel with good scaling on 8 nodes.
>>>>>
>>>>
>>>> If you look at the STREAMS numbers, you can see that your system is
>>>> only able to support about 2 cores with the
>>>> available memory bandwidth. Thus for bandwidth constrained operations
>>>> (almost everything in sparse linear algebra
>>>> and solvers), your speedup will not be bigger than 2.
>>>>
>>>> Other codes may do well on this machine, but they would be compute
>>>> constrained, using things like DGEMM.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Kindly let me know.
>>>>>
>>>>> Venkatesh
>>>>>
>>>>>
>>>>>
>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>
>>>>>>
>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hi,
>>>>>> > I have emailed the mumps-user list.
>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>> scale well.
>>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>>> time.
>>>>>> >
>>>>>> > So is there another way ?
>>>>>> >
>>>>>> > cheers,
>>>>>> > Venkatesh
>>>>>> >
>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>> cores since it is failing with small value of icntl(14).
>>>>>> >
>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>> -mat_mumps_icntl_14 200)
>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>> -mat_mumps_icntl_14 85  )
>>>>>> >
>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that is
>>>>>> a relatively small effect
>>>>>> >
>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward solve.
>>>>>> You might try emailing
>>>>>> > the list for them. However, I would bet that your system has enough
>>>>>> bandwidth for 2 procs
>>>>>> > and not much more.
>>>>>> >
>>>>>> >   Thanks,
>>>>>> >
>>>>>> >      Matt
>>>>>> >
>>>>>> > Venkatesh
>>>>>> >
>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>> >
>>>>>> > It works. However, the problem is, if I submit in 1 core I get the
>>>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>>>> 3500secs.
>>>>>> >
>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>> Otherwise we are just guessing.
>>>>>> >
>>>>>> >     Matt
>>>>>> >
>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>> >
>>>>>> > Kindly let me know.
>>>>>> >
>>>>>> > Venkatesh
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>> david.knezevic at akselos.com> wrote:
>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>> > Hi,
>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>> >
>>>>>> > A and B are of sizes 3600x3600
>>>>>> >
>>>>>> > I run with this command :
>>>>>> >
>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>> >
>>>>>> > I get this error: (I get result only when I give 1 or 2 processors)
>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>> ------------------------------------
>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>>> >
>>>>>> >
>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>> as:
>>>>>> >
>>>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2) is
>>>>>> positive, then the number of entries that are missing in S at the moment
>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>> ICNTL(23) should be increased."
>>>>>> >
>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>> space size:
>>>>>> >
>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and the
>>>>>> factorization phases. It corresponds to the percentage increase in the
>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>> >
>>>>>> > So, for example, you can avoid this error via the following command
>>>>>> line argument to PETSc: "-mat_mumps_icntl_14 30", where 30 indicates that
>>>>>> we allow a 30% increase in the workspace instead of the default 20%.
>>>>>> >
>>>>>> > David
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > -- Norbert Wiener
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > -- Norbert Wiener
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/cbe3b2b4/attachment-0001.html>

From venkateshgk.j at gmail.com  Sat May 23 07:09:24 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sat, 23 May 2015 17:39:24 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
Message-ID: <CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>

Hi,
Thanks.
Per node it has 24 cores and each core has 4 GB RAM. And the job was
submitted in 10 nodes.

So, does it mean it requires 10G for one core ? or for 1 node ?

cheers,

Venkatesh

On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi,
>> The same eigenproblem runs with 120 GB RAM in a serial machine in Matlab.
>>
>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right ?
>>
>
> I do not know how MUMPS allocates memory, but the message is unambiguous.
> Also,
> this is concerned with the memory available per node. Do you know how many
> processes
> per node were scheduled? The message below indicates that it was trying to
> allocate 10G
> for one process.
>
>
>> And for small matrices it is having negative scaling i.e 24 core is
>> running faster.
>>
>
> Yes, for strong scaling you always get slowdown eventually since overheads
> dominate
> work, see Amdahl's Law.
>
>   Thanks,
>
>      Matt
>
>
>> I have attached the submission script.
>>
>> Pls see.. Kindly let me know
>>
>> cheers,
>> Venkatesh
>>
>>
>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi again,
>>>>
>>>> I have installed the Petsc and Slepc in Cray with intel compilers with
>>>> Mumps.
>>>>
>>>> I am getting this error when I solve eigenvalue problem with large
>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>
>>>
>>> It ran out of memory on the node.
>>>
>>>
>>>> Also it is again not scaling well for small matrices.
>>>>
>>>
>>> MUMPS strong scaling for small matrices is not very good. Weak scaling
>>> is looking at big matrices.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Kindly let me know what to do.
>>>>
>>>> cheers,
>>>>
>>>> Venkatesh
>>>>
>>>>
>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have attached the log of the command which I gave in the master
>>>>>> node: make streams NPMAX=32
>>>>>>
>>>>>> I dont know why it says 'It appears you have only 1 node'. But other
>>>>>> codes run in parallel with good scaling on 8 nodes.
>>>>>>
>>>>>
>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>> only able to support about 2 cores with the
>>>>> available memory bandwidth. Thus for bandwidth constrained operations
>>>>> (almost everything in sparse linear algebra
>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>
>>>>> Other codes may do well on this machine, but they would be compute
>>>>> constrained, using things like DGEMM.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Kindly let me know.
>>>>>>
>>>>>> Venkatesh
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>
>>>>>>>
>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi,
>>>>>>> > I have emailed the mumps-user list.
>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>> scale well.
>>>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>>>> time.
>>>>>>> >
>>>>>>> > So is there another way ?
>>>>>>> >
>>>>>>> > cheers,
>>>>>>> > Venkatesh
>>>>>>> >
>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>> knepley at gmail.com> wrote:
>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>> >
>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>> >
>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that
>>>>>>> is a relatively small effect
>>>>>>> >
>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward solve.
>>>>>>> You might try emailing
>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>> enough bandwidth for 2 procs
>>>>>>> > and not much more.
>>>>>>> >
>>>>>>> >   Thanks,
>>>>>>> >
>>>>>>> >      Matt
>>>>>>> >
>>>>>>> > Venkatesh
>>>>>>> >
>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>> knepley at gmail.com> wrote:
>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>>> >
>>>>>>> > It works. However, the problem is, if I submit in 1 core I get the
>>>>>>> answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it takes
>>>>>>> 3500secs.
>>>>>>> >
>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>> Otherwise we are just guessing.
>>>>>>> >
>>>>>>> >     Matt
>>>>>>> >
>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>> >
>>>>>>> > Kindly let me know.
>>>>>>> >
>>>>>>> > Venkatesh
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>> > Hi,
>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>> >
>>>>>>> > A and B are of sizes 3600x3600
>>>>>>> >
>>>>>>> > I run with this command :
>>>>>>> >
>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>> >
>>>>>>> > I get this error: (I get result only when I give 1 or 2 processors)
>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>> ------------------------------------
>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>>>>>> phase: INFO(1)=-9, INFO(2)=2024
>>>>>>> >
>>>>>>> >
>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>> as:
>>>>>>> >
>>>>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2)
>>>>>>> is positive, then the number of entries that are missing in S at the moment
>>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>>> ICNTL(23) should be increased."
>>>>>>> >
>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>> space size:
>>>>>>> >
>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>> >
>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>> default 20%.
>>>>>>> >
>>>>>>> > David
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > -- Norbert Wiener
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > -- Norbert Wiener
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/b8e72e08/attachment.html>

From knepley at gmail.com  Sat May 23 07:13:27 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 23 May 2015 07:13:27 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
Message-ID: <CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>

On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
> Thanks.
> Per node it has 24 cores and each core has 4 GB RAM. And the job was
> submitted in 10 nodes.
>
> So, does it mean it requires 10G for one core ? or for 1 node ?
>

The error message from MUMPS said that it tried to allocate 10G. We must
assume each process
tried to do the same thing. That means if you scheduled 24 processes on a
node, it would try to
allocate at least 240G, which is in excess of what you specify above.

Note that this has nothing to do with PETSc. It is all in the documentation
for that machine and its
scheduling policy.

  Thanks,

     Matt


> cheers,
>
> Venkatesh
>
> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>> Matlab.
>>>
>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right
>>> ?
>>>
>>
>> I do not know how MUMPS allocates memory, but the message is unambiguous.
>> Also,
>> this is concerned with the memory available per node. Do you know how
>> many processes
>> per node were scheduled? The message below indicates that it was trying
>> to allocate 10G
>> for one process.
>>
>>
>>> And for small matrices it is having negative scaling i.e 24 core is
>>> running faster.
>>>
>>
>> Yes, for strong scaling you always get slowdown eventually since
>> overheads dominate
>> work, see Amdahl's Law.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> I have attached the submission script.
>>>
>>> Pls see.. Kindly let me know
>>>
>>> cheers,
>>> Venkatesh
>>>
>>>
>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi again,
>>>>>
>>>>> I have installed the Petsc and Slepc in Cray with intel compilers with
>>>>> Mumps.
>>>>>
>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>
>>>>
>>>> It ran out of memory on the node.
>>>>
>>>>
>>>>> Also it is again not scaling well for small matrices.
>>>>>
>>>>
>>>> MUMPS strong scaling for small matrices is not very good. Weak scaling
>>>> is looking at big matrices.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Kindly let me know what to do.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Venkatesh
>>>>>
>>>>>
>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>> node: make streams NPMAX=32
>>>>>>>
>>>>>>> I dont know why it says 'It appears you have only 1 node'. But other
>>>>>>> codes run in parallel with good scaling on 8 nodes.
>>>>>>>
>>>>>>
>>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>>> only able to support about 2 cores with the
>>>>>> available memory bandwidth. Thus for bandwidth constrained operations
>>>>>> (almost everything in sparse linear algebra
>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>
>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>> constrained, using things like DGEMM.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> Kindly let me know.
>>>>>>>
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>
>>>>>>>>
>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > Hi,
>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>>> scale well.
>>>>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>>>>> time.
>>>>>>>> >
>>>>>>>> > So is there another way ?
>>>>>>>> >
>>>>>>>> > cheers,
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>>> >
>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>> >
>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that
>>>>>>>> is a relatively small effect
>>>>>>>> >
>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>> solve. You might try emailing
>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>> enough bandwidth for 2 procs
>>>>>>>> > and not much more.
>>>>>>>> >
>>>>>>>> >   Thanks,
>>>>>>>> >
>>>>>>>> >      Matt
>>>>>>>> >
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>>>> >
>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>> takes 3500secs.
>>>>>>>> >
>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>> Otherwise we are just guessing.
>>>>>>>> >
>>>>>>>> >     Matt
>>>>>>>> >
>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>>> >
>>>>>>>> > Kindly let me know.
>>>>>>>> >
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi,
>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>> >
>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>> >
>>>>>>>> > I run with this command :
>>>>>>>> >
>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>> >
>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>> processors)
>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> ------------------------------------
>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>> as:
>>>>>>>> >
>>>>>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2)
>>>>>>>> is positive, then the number of entries that are missing in S at the moment
>>>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>>>> ICNTL(23) should be increased."
>>>>>>>> >
>>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>>> space size:
>>>>>>>> >
>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>> >
>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>> default 20%.
>>>>>>>> >
>>>>>>>> > David
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/afcc901f/attachment-0001.html>

From vijay.gopal.c at gmail.com  Sat May 23 12:27:56 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Sat, 23 May 2015 19:27:56 +0200
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
Message-ID: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>

Hi,

I'm having problems calculating large number of eigenvalues(nev>400) with
SLEPc, i get perfect results with small number of eigenvalues.


*PETSc: Branch: origin/maint ; commit: 2b04bc0*
*SLEPc: Branch: origin/maint ; commit: e1f03d9*


This is the error message i get:

*Intel Parallel Studio XE 2013 loaded*
*Intel(R) MPI Library 4.1 (4.1.3.049) loaded*
*[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------*
*[0]PETSC ERROR: Invalid argument*
*[0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 *
*[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
<http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.*
*[0]PETSC ERROR: Petsc Release Version 3.5.3, unknown?*
*[0]PETSC ERROR: ./ex1 on a arch-linux2-c-opt named eoscomp7 by vijayc Sat
May 23 19:04:55 2015*
*[0]PETSC ERROR: Configure options --with-64-bit-indices --with-cc=mpiicc
--with-fc=mpiifort --with-cxx=mpiicpc --with-debugging=1 --FOPTFLAGS="-O3
-xAV*
*X -fno-alias -no-prec-div -no-prec-sqrt -ip " --COPTFLAGS="-O3 -xAVX
-fno-alias -no-prec-div -no-prec-sqrt -ip " --CXXOPTFLAGS="-O3 -xAVX
-fno-alias -no-prec-div -no-prec-sqrt -ip " --download-fblaslapack
--with-x=false*
*[0]PETSC ERROR: #1 BVScaleColumn() line 380 in
/eos1/p1517/vijayc/slepc_basic/src/sys/classes/bv/interface/bvops.c*
*[0]PETSC ERROR: #2 EPSFullLanczos() line 200 in
/eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylov.c*
*[0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 56 in
/eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylovschur/ks-symm.c*
*[0]PETSC ERROR: #4 EPSSolve() line 99 in
/eos1/p1517/vijayc/slepc_basic/src/eps/interface/epssolve.c*
*[0]PETSC ERROR: #5 main() line 143 in
/users/p1517/vijayc/modelization/ntrou3/sites_18/obc_10/isz_0/ex1.c*
*[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------*
*application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0*

thanks a lot,
 Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/db4fcff0/attachment.html>

From bsmith at mcs.anl.gov  Sat May 23 14:04:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 23 May 2015 14:04:28 -0500
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
In-Reply-To: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
References: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
Message-ID: <C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>


   This error message is almost always because a Nan or Inf has crept into the numbers and then gotten into a norm or inner product. 
You can run with -fp_trap or run a debugger and have it catch floating point exceptions to see the first point where the Nan or Inf appeared and help track down the cause.  But do all this tracking down using a debug version of the library without all the extra optimization flags you have put into the COPTFLAGS flags etc.

  Barry

> On May 23, 2015, at 12:27 PM, Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> wrote:
> 
> Hi,
> 
> I'm having problems calculating large number of eigenvalues(nev>400) with SLEPc, i get perfect results with small number of eigenvalues.
> 
> 
> PETSc: Branch: origin/maint ; commit: 2b04bc0
> SLEPc: Branch: origin/maint ; commit: e1f03d9
> 
> This is the error message i get:
> 
> Intel Parallel Studio XE 2013 loaded
> Intel(R) MPI Library 4.1 (4.1.3.049) loaded
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown?
> [0]PETSC ERROR: ./ex1 on a arch-linux2-c-opt named eoscomp7 by vijayc Sat May 23 19:04:55 2015
> [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cc=mpiicc --with-fc=mpiifort --with-cxx=mpiicpc --with-debugging=1 --FOPTFLAGS="-O3 -xAV
> X -fno-alias -no-prec-div -no-prec-sqrt -ip " --COPTFLAGS="-O3 -xAVX -fno-alias -no-prec-div -no-prec-sqrt -ip " --CXXOPTFLAGS="-O3 -xAVX -fno-alias -no-prec-div -no-prec-sqrt -ip " --download-fblaslapack --with-x=false
> [0]PETSC ERROR: #1 BVScaleColumn() line 380 in /eos1/p1517/vijayc/slepc_basic/src/sys/classes/bv/interface/bvops.c
> [0]PETSC ERROR: #2 EPSFullLanczos() line 200 in /eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylov.c
> [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 56 in /eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylovschur/ks-symm.c
> [0]PETSC ERROR: #4 EPSSolve() line 99 in /eos1/p1517/vijayc/slepc_basic/src/eps/interface/epssolve.c
> [0]PETSC ERROR: #5 main() line 143 in /users/p1517/vijayc/modelization/ntrou3/sites_18/obc_10/isz_0/ex1.c
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
> 
> thanks a lot,
>  Vijay


From venkateshgk.j at gmail.com  Sun May 24 08:57:38 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sun, 24 May 2015 19:27:38 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
	<CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
Message-ID: <CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>

I am using Matload option as in the ex7.c code given by the Slepc.
ierr = MatLoad(A,viewer);CHKERRQ(ierr);


There is no problem here right ? or any additional option is required for
very large matrices while running the eigensolver in parallel ?

cheers,
Venkatesh

On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi,
>> Thanks.
>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>> submitted in 10 nodes.
>>
>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>
>
> The error message from MUMPS said that it tried to allocate 10G. We must
> assume each process
> tried to do the same thing. That means if you scheduled 24 processes on a
> node, it would try to
> allocate at least 240G, which is in excess of what you specify above.
>
> Note that this has nothing to do with PETSc. It is all in the
> documentation for that machine and its
> scheduling policy.
>
>   Thanks,
>
>      Matt
>
>
>> cheers,
>>
>> Venkatesh
>>
>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>> Matlab.
>>>>
>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right
>>>> ?
>>>>
>>>
>>> I do not know how MUMPS allocates memory, but the message is
>>> unambiguous. Also,
>>> this is concerned with the memory available per node. Do you know how
>>> many processes
>>> per node were scheduled? The message below indicates that it was trying
>>> to allocate 10G
>>> for one process.
>>>
>>>
>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>> running faster.
>>>>
>>>
>>> Yes, for strong scaling you always get slowdown eventually since
>>> overheads dominate
>>> work, see Amdahl's Law.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> I have attached the submission script.
>>>>
>>>> Pls see.. Kindly let me know
>>>>
>>>> cheers,
>>>> Venkatesh
>>>>
>>>>
>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi again,
>>>>>>
>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>> with Mumps.
>>>>>>
>>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>
>>>>>
>>>>> It ran out of memory on the node.
>>>>>
>>>>>
>>>>>> Also it is again not scaling well for small matrices.
>>>>>>
>>>>>
>>>>> MUMPS strong scaling for small matrices is not very good. Weak scaling
>>>>> is looking at big matrices.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Kindly let me know what to do.
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Venkatesh
>>>>>>
>>>>>>
>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>>> node: make streams NPMAX=32
>>>>>>>>
>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>
>>>>>>>
>>>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>>>> only able to support about 2 cores with the
>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>
>>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>>> constrained, using things like DGEMM.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Kindly let me know.
>>>>>>>>
>>>>>>>> Venkatesh
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi,
>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>>>> scale well.
>>>>>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>>>>>> time.
>>>>>>>>> >
>>>>>>>>> > So is there another way ?
>>>>>>>>> >
>>>>>>>>> > cheers,
>>>>>>>>> > Venkatesh
>>>>>>>>> >
>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>>>> >
>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>>> >
>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that
>>>>>>>>> is a relatively small effect
>>>>>>>>> >
>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>> solve. You might try emailing
>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>> > and not much more.
>>>>>>>>> >
>>>>>>>>> >   Thanks,
>>>>>>>>> >
>>>>>>>>> >      Matt
>>>>>>>>> >
>>>>>>>>> > Venkatesh
>>>>>>>>> >
>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>>>>> >
>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>>> takes 3500secs.
>>>>>>>>> >
>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>> >
>>>>>>>>> >     Matt
>>>>>>>>> >
>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev
>>>>>>>>> 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>>>> >
>>>>>>>>> > Kindly let me know.
>>>>>>>>> >
>>>>>>>>> > Venkatesh
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>> > Hi,
>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>> >
>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>> >
>>>>>>>>> > I run with this command :
>>>>>>>>> >
>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>> >
>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>> processors)
>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>> ------------------------------------
>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>>> as:
>>>>>>>>> >
>>>>>>>>> > "?9 Main internal real/complex workarray S too small. If INFO(2)
>>>>>>>>> is positive, then the number of entries that are missing in S at the moment
>>>>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>>>>> then its absolute value should be multiplied by 1 million. If an error ?9
>>>>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>>>>> ICNTL(23) should be increased."
>>>>>>>>> >
>>>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>>>> space size:
>>>>>>>>> >
>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>> >
>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>> default 20%.
>>>>>>>>> >
>>>>>>>>> > David
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> > -- Norbert Wiener
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> > -- Norbert Wiener
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150524/06139bbe/attachment-0001.html>

From knepley at gmail.com  Sun May 24 09:06:21 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 24 May 2015 09:06:21 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
	<CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
	<CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>
Message-ID: <CAMYG4Gn=HA6VFY79fmJW-0uAoL+LWi8TEbZYGwf0Nw+QKAZp8A@mail.gmail.com>

On Sun, May 24, 2015 at 8:57 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> I am using Matload option as in the ex7.c code given by the Slepc.
> ierr = MatLoad(A,viewer);CHKERRQ(ierr);
>
>
> There is no problem here right ? or any additional option is required for
> very large matrices while running the eigensolver in parallel ?
>

This will load the matrix from the viewer (presumably disk). There are no
options for large matrices.

  Thanks,

      Matt


> cheers,
> Venkatesh
>
> On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>> Thanks.
>>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>>> submitted in 10 nodes.
>>>
>>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>>
>>
>> The error message from MUMPS said that it tried to allocate 10G. We must
>> assume each process
>> tried to do the same thing. That means if you scheduled 24 processes on a
>> node, it would try to
>> allocate at least 240G, which is in excess of what you specify above.
>>
>> Note that this has nothing to do with PETSc. It is all in the
>> documentation for that machine and its
>> scheduling policy.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> cheers,
>>>
>>> Venkatesh
>>>
>>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>>> Matlab.
>>>>>
>>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in
>>>>> right ?
>>>>>
>>>>
>>>> I do not know how MUMPS allocates memory, but the message is
>>>> unambiguous. Also,
>>>> this is concerned with the memory available per node. Do you know how
>>>> many processes
>>>> per node were scheduled? The message below indicates that it was trying
>>>> to allocate 10G
>>>> for one process.
>>>>
>>>>
>>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>>> running faster.
>>>>>
>>>>
>>>> Yes, for strong scaling you always get slowdown eventually since
>>>> overheads dominate
>>>> work, see Amdahl's Law.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> I have attached the submission script.
>>>>>
>>>>> Pls see.. Kindly let me know
>>>>>
>>>>> cheers,
>>>>> Venkatesh
>>>>>
>>>>>
>>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi again,
>>>>>>>
>>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>>> with Mumps.
>>>>>>>
>>>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>>
>>>>>>
>>>>>> It ran out of memory on the node.
>>>>>>
>>>>>>
>>>>>>> Also it is again not scaling well for small matrices.
>>>>>>>
>>>>>>
>>>>>> MUMPS strong scaling for small matrices is not very good. Weak
>>>>>> scaling is looking at big matrices.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> Kindly let me know what to do.
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>>>> node: make streams NPMAX=32
>>>>>>>>>
>>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>>
>>>>>>>>
>>>>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>>>>> only able to support about 2 cores with the
>>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>>
>>>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>>>> constrained, using things like DGEMM.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Kindly let me know.
>>>>>>>>>
>>>>>>>>> Venkatesh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Hi,
>>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>>>>> scale well.
>>>>>>>>>> > I wanted to ask if this job takes much time, then if I submit
>>>>>>>>>> on more cores, I have to increase the icntl(14).. which would again take
>>>>>>>>>> long time.
>>>>>>>>>> >
>>>>>>>>>> > So is there another way ?
>>>>>>>>>> >
>>>>>>>>>> > cheers,
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>>>>> >
>>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>>>> >
>>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but
>>>>>>>>>> that is a relatively small effect
>>>>>>>>>> >
>>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>>> solve. You might try emailing
>>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>>> > and not much more.
>>>>>>>>>> >
>>>>>>>>>> >   Thanks,
>>>>>>>>>> >
>>>>>>>>>> >      Matt
>>>>>>>>>> >
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi, Thanks for the information. I now increased the workspace
>>>>>>>>>> by adding '-mat_mumps_icntl_14 100'
>>>>>>>>>> >
>>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>>>> takes 3500secs.
>>>>>>>>>> >
>>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>>> >
>>>>>>>>>> >     Matt
>>>>>>>>>> >
>>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev
>>>>>>>>>> 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>>>>> >
>>>>>>>>>> > Kindly let me know.
>>>>>>>>>> >
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi,
>>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>>> >
>>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>>> >
>>>>>>>>>> > I run with this command :
>>>>>>>>>> >
>>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>>> >
>>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>>> processors)
>>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>> ------------------------------------
>>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>>>> as:
>>>>>>>>>> >
>>>>>>>>>> > "?9 Main internal real/complex workarray S too small. If
>>>>>>>>>> INFO(2) is positive, then the number of entries that are missing in S at
>>>>>>>>>> the moment when the error is raised is available in INFO(2). If INFO(2) is
>>>>>>>>>> negative, then its absolute value should be multiplied by 1 million. If an
>>>>>>>>>> error ?9 occurs, the user should increase the value of ICNTL(14) before
>>>>>>>>>> calling the factorization (JOB=2) again, except if ICNTL(23) is provided,
>>>>>>>>>> in which case ICNTL(23) should be increased."
>>>>>>>>>> >
>>>>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>>>>> space size:
>>>>>>>>>> >
>>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>>> >
>>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>>> default 20%.
>>>>>>>>>> >
>>>>>>>>>> > David
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150524/27265309/attachment.html>

From salazardetroya at gmail.com  Sun May 24 18:01:21 2015
From: salazardetroya at gmail.com (Miguel Angel Salazar de Troya)
Date: Sun, 24 May 2015 18:01:21 -0500
Subject: [petsc-users] Passing a context to TSPreStep()
Message-ID: <CACH89COUWDLDG2sj128KWG45=-L6ofrcCqkvV0mNNj4XfYFdMw@mail.gmail.com>

Hi all

I would like to record the information about the failed steps in an
adaptive time stepping algorithm. I have noticed that I could call
TSPreStep() and get the current time step, but I want to store it in an
external data structure. Now, for TSMonitor(), one of the arguments is a
context that I can use to store some results. Is there a similar thing for
TSPreStep() How could I implement it?

Thanks
Miguel

-- 
*Miguel Angel Salazar de Troya*
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360
salaza11 at illinois.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150524/ac107fda/attachment.html>

From jed at jedbrown.org  Sun May 24 18:07:40 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 24 May 2015 17:07:40 -0600
Subject: [petsc-users] Passing a context to TSPreStep()
In-Reply-To: <CACH89COUWDLDG2sj128KWG45=-L6ofrcCqkvV0mNNj4XfYFdMw@mail.gmail.com>
References: <CACH89COUWDLDG2sj128KWG45=-L6ofrcCqkvV0mNNj4XfYFdMw@mail.gmail.com>
Message-ID: <874mn1shfn.fsf@jedbrown.org>

Miguel Angel Salazar de Troya <salazardetroya at gmail.com> writes:
> I would like to record the information about the failed steps in an
> adaptive time stepping algorithm. I have noticed that I could call
> TSPreStep() and get the current time step, but I want to store it in an
> external data structure. Now, for TSMonitor(), one of the arguments is a
> context that I can use to store some results. Is there a similar thing for
> TSPreStep() How could I implement it?

You could use TSSetApplicationContext and TSGetApplicationContext.  I
think it is purely historical that TSSetPreStep() does not include a
context -- perhaps it should be changed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150524/ae04fa4b/attachment.pgp>

From venkateshgk.j at gmail.com  Mon May 25 02:54:01 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Mon, 25 May 2015 13:24:01 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4Gn=HA6VFY79fmJW-0uAoL+LWi8TEbZYGwf0Nw+QKAZp8A@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
	<CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
	<CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>
	<CAMYG4Gn=HA6VFY79fmJW-0uAoL+LWi8TEbZYGwf0Nw+QKAZp8A@mail.gmail.com>
Message-ID: <CABzT7hONOW+hdht5FFip6SsJrsySbHe_=gkMEdg3KWLEnMx05A@mail.gmail.com>

Ok this will load the matrices in parallel correct ?

On Sun, May 24, 2015 at 7:36 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, May 24, 2015 at 8:57 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> I am using Matload option as in the ex7.c code given by the Slepc.
>> ierr = MatLoad(A,viewer);CHKERRQ(ierr);
>>
>>
>> There is no problem here right ? or any additional option is required for
>> very large matrices while running the eigensolver in parallel ?
>>
>
> This will load the matrix from the viewer (presumably disk). There are no
> options for large matrices.
>
>   Thanks,
>
>       Matt
>
>
>> cheers,
>> Venkatesh
>>
>> On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Thanks.
>>>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>>>> submitted in 10 nodes.
>>>>
>>>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>>>
>>>
>>> The error message from MUMPS said that it tried to allocate 10G. We must
>>> assume each process
>>> tried to do the same thing. That means if you scheduled 24 processes on
>>> a node, it would try to
>>> allocate at least 240G, which is in excess of what you specify above.
>>>
>>> Note that this has nothing to do with PETSc. It is all in the
>>> documentation for that machine and its
>>> scheduling policy.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> cheers,
>>>>
>>>> Venkatesh
>>>>
>>>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>>>> Matlab.
>>>>>>
>>>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in
>>>>>> right ?
>>>>>>
>>>>>
>>>>> I do not know how MUMPS allocates memory, but the message is
>>>>> unambiguous. Also,
>>>>> this is concerned with the memory available per node. Do you know how
>>>>> many processes
>>>>> per node were scheduled? The message below indicates that it was
>>>>> trying to allocate 10G
>>>>> for one process.
>>>>>
>>>>>
>>>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>>>> running faster.
>>>>>>
>>>>>
>>>>> Yes, for strong scaling you always get slowdown eventually since
>>>>> overheads dominate
>>>>> work, see Amdahl's Law.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> I have attached the submission script.
>>>>>>
>>>>>> Pls see.. Kindly let me know
>>>>>>
>>>>>> cheers,
>>>>>> Venkatesh
>>>>>>
>>>>>>
>>>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi again,
>>>>>>>>
>>>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>>>> with Mumps.
>>>>>>>>
>>>>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>>>
>>>>>>>
>>>>>>> It ran out of memory on the node.
>>>>>>>
>>>>>>>
>>>>>>>> Also it is again not scaling well for small matrices.
>>>>>>>>
>>>>>>>
>>>>>>> MUMPS strong scaling for small matrices is not very good. Weak
>>>>>>> scaling is looking at big matrices.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Kindly let me know what to do.
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Venkatesh
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>>>>> node: make streams NPMAX=32
>>>>>>>>>>
>>>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you look at the STREAMS numbers, you can see that your system
>>>>>>>>> is only able to support about 2 cores with the
>>>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>>>
>>>>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>>>>> constrained, using things like DGEMM.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Kindly let me know.
>>>>>>>>>>
>>>>>>>>>> Venkatesh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    Run the streams benchmark on this system and send the
>>>>>>>>>>> results.
>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Hi,
>>>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other
>>>>>>>>>>> codes scale well.
>>>>>>>>>>> > I wanted to ask if this job takes much time, then if I submit
>>>>>>>>>>> on more cores, I have to increase the icntl(14).. which would again take
>>>>>>>>>>> long time.
>>>>>>>>>>> >
>>>>>>>>>>> > So is there another way ?
>>>>>>>>>>> >
>>>>>>>>>>> > cheers,
>>>>>>>>>>> > Venkatesh
>>>>>>>>>>> >
>>>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on
>>>>>>>>>>> different processors. I had to increase the workspace icntl(14) when I
>>>>>>>>>>> submit on more cores since it is failing with small value of icntl(14).
>>>>>>>>>>> >
>>>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>>>>> >
>>>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but
>>>>>>>>>>> that is a relatively small effect
>>>>>>>>>>> >
>>>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>>>> solve. You might try emailing
>>>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>>>> > and not much more.
>>>>>>>>>>> >
>>>>>>>>>>> >   Thanks,
>>>>>>>>>>> >
>>>>>>>>>>> >      Matt
>>>>>>>>>>> >
>>>>>>>>>>> > Venkatesh
>>>>>>>>>>> >
>>>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>> > Hi, Thanks for the information. I now increased the workspace
>>>>>>>>>>> by adding '-mat_mumps_icntl_14 100'
>>>>>>>>>>> >
>>>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>>>>> takes 3500secs.
>>>>>>>>>>> >
>>>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>>>> >
>>>>>>>>>>> >     Matt
>>>>>>>>>>> >
>>>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2
>>>>>>>>>>> -eps_nev 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly
>>>>>>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>>>>>>> 100'
>>>>>>>>>>> >
>>>>>>>>>>> > Kindly let me know.
>>>>>>>>>>> >
>>>>>>>>>>> > Venkatesh
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>> > Hi,
>>>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>>>> >
>>>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>>>> >
>>>>>>>>>>> > I run with this command :
>>>>>>>>>>> >
>>>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>>>> >
>>>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>>>> processors)
>>>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>> ------------------------------------
>>>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>>>>> as:
>>>>>>>>>>> >
>>>>>>>>>>> > "?9 Main internal real/complex workarray S too small. If
>>>>>>>>>>> INFO(2) is positive, then the number of entries that are missing in S at
>>>>>>>>>>> the moment when the error is raised is available in INFO(2). If INFO(2) is
>>>>>>>>>>> negative, then its absolute value should be multiplied by 1 million. If an
>>>>>>>>>>> error ?9 occurs, the user should increase the value of ICNTL(14) before
>>>>>>>>>>> calling the factorization (JOB=2) again, except if ICNTL(23) is provided,
>>>>>>>>>>> in which case ICNTL(23) should be increased."
>>>>>>>>>>> >
>>>>>>>>>>> > This says that you should use ICTNL(14) to increase the
>>>>>>>>>>> working space size:
>>>>>>>>>>> >
>>>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis
>>>>>>>>>>> and the factorization phases. It corresponds to the percentage increase in
>>>>>>>>>>> the estimated working space. When significant extra fill-in is caused by
>>>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>>>> >
>>>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>>>> default 20%.
>>>>>>>>>>> >
>>>>>>>>>>> > David
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>> their experiments lead.
>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>> their experiments lead.
>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150525/4f2765bc/attachment-0001.html>

From knepley at gmail.com  Mon May 25 04:06:30 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 25 May 2015 04:06:30 -0500
Subject: [petsc-users] MUMPS error
In-Reply-To: <CABzT7hONOW+hdht5FFip6SsJrsySbHe_=gkMEdg3KWLEnMx05A@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
	<CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
	<CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>
	<CAMYG4Gn=HA6VFY79fmJW-0uAoL+LWi8TEbZYGwf0Nw+QKAZp8A@mail.gmail.com>
	<CABzT7hONOW+hdht5FFip6SsJrsySbHe_=gkMEdg3KWLEnMx05A@mail.gmail.com>
Message-ID: <CAMYG4G=Y_Tgi_OyQR36+ABRS1H76jXono1b0-tBqD3nOf5t8jQ@mail.gmail.com>

On Mon, May 25, 2015 at 2:54 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Ok this will load the matrices in parallel correct ?
>

Yes

   Matt


> On Sun, May 24, 2015 at 7:36 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sun, May 24, 2015 at 8:57 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> I am using Matload option as in the ex7.c code given by the Slepc.
>>> ierr = MatLoad(A,viewer);CHKERRQ(ierr);
>>>
>>>
>>> There is no problem here right ? or any additional option is required
>>> for very large matrices while running the eigensolver in parallel ?
>>>
>>
>> This will load the matrix from the viewer (presumably disk). There are no
>> options for large matrices.
>>
>>   Thanks,
>>
>>       Matt
>>
>>
>>> cheers,
>>> Venkatesh
>>>
>>> On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Thanks.
>>>>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>>>>> submitted in 10 nodes.
>>>>>
>>>>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>>>>
>>>>
>>>> The error message from MUMPS said that it tried to allocate 10G. We
>>>> must assume each process
>>>> tried to do the same thing. That means if you scheduled 24 processes on
>>>> a node, it would try to
>>>> allocate at least 240G, which is in excess of what you specify above.
>>>>
>>>> Note that this has nothing to do with PETSc. It is all in the
>>>> documentation for that machine and its
>>>> scheduling policy.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> cheers,
>>>>>
>>>>> Venkatesh
>>>>>
>>>>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>>>>> Matlab.
>>>>>>>
>>>>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in
>>>>>>> right ?
>>>>>>>
>>>>>>
>>>>>> I do not know how MUMPS allocates memory, but the message is
>>>>>> unambiguous. Also,
>>>>>> this is concerned with the memory available per node. Do you know how
>>>>>> many processes
>>>>>> per node were scheduled? The message below indicates that it was
>>>>>> trying to allocate 10G
>>>>>> for one process.
>>>>>>
>>>>>>
>>>>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>>>>> running faster.
>>>>>>>
>>>>>>
>>>>>> Yes, for strong scaling you always get slowdown eventually since
>>>>>> overheads dominate
>>>>>> work, see Amdahl's Law.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> I have attached the submission script.
>>>>>>>
>>>>>>> Pls see.. Kindly let me know
>>>>>>>
>>>>>>> cheers,
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi again,
>>>>>>>>>
>>>>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>>>>> with Mumps.
>>>>>>>>>
>>>>>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>>>>
>>>>>>>>
>>>>>>>> It ran out of memory on the node.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also it is again not scaling well for small matrices.
>>>>>>>>>
>>>>>>>>
>>>>>>>> MUMPS strong scaling for small matrices is not very good. Weak
>>>>>>>> scaling is looking at big matrices.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>      Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Kindly let me know what to do.
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Venkatesh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have attached the log of the command which I gave in the
>>>>>>>>>>> master node: make streams NPMAX=32
>>>>>>>>>>>
>>>>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If you look at the STREAMS numbers, you can see that your system
>>>>>>>>>> is only able to support about 2 cores with the
>>>>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>>>>
>>>>>>>>>> Other codes may do well on this machine, but they would be
>>>>>>>>>> compute constrained, using things like DGEMM.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Kindly let me know.
>>>>>>>>>>>
>>>>>>>>>>> Venkatesh
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <
>>>>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>    Run the streams benchmark on this system and send the
>>>>>>>>>>>> results.
>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Hi,
>>>>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other
>>>>>>>>>>>> codes scale well.
>>>>>>>>>>>> > I wanted to ask if this job takes much time, then if I submit
>>>>>>>>>>>> on more cores, I have to increase the icntl(14).. which would again take
>>>>>>>>>>>> long time.
>>>>>>>>>>>> >
>>>>>>>>>>>> > So is there another way ?
>>>>>>>>>>>> >
>>>>>>>>>>>> > cheers,
>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on
>>>>>>>>>>>> different processors. I had to increase the workspace icntl(14) when I
>>>>>>>>>>>> submit on more cores since it is failing with small value of icntl(14).
>>>>>>>>>>>> >
>>>>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>>>>>> >
>>>>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but
>>>>>>>>>>>> that is a relatively small effect
>>>>>>>>>>>> >
>>>>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>>>>> solve. You might try emailing
>>>>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>>>>> > and not much more.
>>>>>>>>>>>> >
>>>>>>>>>>>> >   Thanks,
>>>>>>>>>>>> >
>>>>>>>>>>>> >      Matt
>>>>>>>>>>>> >
>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>> > Hi, Thanks for the information. I now increased the workspace
>>>>>>>>>>>> by adding '-mat_mumps_icntl_14 100'
>>>>>>>>>>>> >
>>>>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I
>>>>>>>>>>>> get the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100'
>>>>>>>>>>>> it takes 3500secs.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>>>>> >
>>>>>>>>>>>> >     Matt
>>>>>>>>>>>> >
>>>>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2
>>>>>>>>>>>> -eps_nev 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly
>>>>>>>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>>>>>>>> 100'
>>>>>>>>>>>> >
>>>>>>>>>>>> > Kindly let me know.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>> > Hi,
>>>>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>>>>> >
>>>>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>>>>> >
>>>>>>>>>>>> > I run with this command :
>>>>>>>>>>>> >
>>>>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type
>>>>>>>>>>>> sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>>>>> >
>>>>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>>>>> processors)
>>>>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>>>>>> as:
>>>>>>>>>>>> >
>>>>>>>>>>>> > "?9 Main internal real/complex workarray S too small. If
>>>>>>>>>>>> INFO(2) is positive, then the number of entries that are missing in S at
>>>>>>>>>>>> the moment when the error is raised is available in INFO(2). If INFO(2) is
>>>>>>>>>>>> negative, then its absolute value should be multiplied by 1 million. If an
>>>>>>>>>>>> error ?9 occurs, the user should increase the value of ICNTL(14) before
>>>>>>>>>>>> calling the factorization (JOB=2) again, except if ICNTL(23) is provided,
>>>>>>>>>>>> in which case ICNTL(23) should be increased."
>>>>>>>>>>>> >
>>>>>>>>>>>> > This says that you should use ICTNL(14) to increase the
>>>>>>>>>>>> working space size:
>>>>>>>>>>>> >
>>>>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis
>>>>>>>>>>>> and the factorization phases. It corresponds to the percentage increase in
>>>>>>>>>>>> the estimated working space. When significant extra fill-in is caused by
>>>>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>>>>> >
>>>>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>>>>> default 20%.
>>>>>>>>>>>> >
>>>>>>>>>>>> > David
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > --
>>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > --
>>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150525/5a640c0b/attachment.html>

From venkateshgk.j at gmail.com  Mon May 25 04:10:02 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Mon, 25 May 2015 14:40:02 +0530
Subject: [petsc-users] MUMPS error
In-Reply-To: <CAMYG4G=Y_Tgi_OyQR36+ABRS1H76jXono1b0-tBqD3nOf5t8jQ@mail.gmail.com>
References: <CABzT7hNhS6uWCsW-zbwJLO-ha9VhE7ZW0dx2n6kha4tqFuuOYg@mail.gmail.com>
	<CAJCWK9BK9qMYXHP7t-v_0P30Y6C4MUh32qG7iT8g6VUGSNE0PA@mail.gmail.com>
	<CABzT7hOq33iOnzdTP2HEsOCjHR3euiMGxtJFpo_BPPCM1Zupeg@mail.gmail.com>
	<CAMYG4G=oSyoq4rTUoUq=DX1TpGFdxsUQKc9o=gAskW4zR+7w1w@mail.gmail.com>
	<CABzT7hPU-sOrQArUddHVqmRs+wh58w7Fw5SnpLEhQhqb+hPEEA@mail.gmail.com>
	<CAMYG4Gn3ckBh+TTrB_M7_u4Mgiy00YgkGwamrhGPqEMhTHvi8w@mail.gmail.com>
	<CABzT7hNCx=vP5DjYH3-5JuJfWu8jV4JemsVaA=rSvC3P-4hLoQ@mail.gmail.com>
	<539F32D8-EFE5-4651-BC3A-E265E0BFECB0@mcs.anl.gov>
	<CABzT7hPzirFDrPJAW96NB5aipaVr8r+pL9OFJGndWXyi9iNwPQ@mail.gmail.com>
	<CAMYG4Gmm5ffSm_MnMT6BSVfZCUW4rHe-uCmJBijVO_KpXYBF6Q@mail.gmail.com>
	<CABzT7hPn969DNw_ru=ZrOqOUFzDsn0xEZgNdYdxfbiEvQP_Qgg@mail.gmail.com>
	<CAMYG4Gna=h+tKHTQP+awdGZa5J+9kEVq5Yb8nkyz5GJjRpndqA@mail.gmail.com>
	<CABzT7hO2UKpsJeRQ71g_2KkQ0w8gV030YQZ9iA1M09-UUXPM1w@mail.gmail.com>
	<CAMYG4GnP55jNKPdeKi0L7dyRqD0Oa-bJDx1=qzYWZi__n3H+Gg@mail.gmail.com>
	<CABzT7hOGboprtbq0+GKcoi_AgtS8kVS1m=YnkWC7Xg9cWwoeKw@mail.gmail.com>
	<CAMYG4G=BHkH7U__-deMjwWFJyregSm1qZQKQgBA+sDbPZuBZiw@mail.gmail.com>
	<CABzT7hOqog8WK_e9_2C261DVWRYtUmqtKv3McpxpJDj0tuCqLA@mail.gmail.com>
	<CAMYG4Gn=HA6VFY79fmJW-0uAoL+LWi8TEbZYGwf0Nw+QKAZp8A@mail.gmail.com>
	<CABzT7hONOW+hdht5FFip6SsJrsySbHe_=gkMEdg3KWLEnMx05A@mail.gmail.com>
	<CAMYG4G=Y_Tgi_OyQR36+ABRS1H76jXono1b0-tBqD3nOf5t8jQ@mail.gmail.com>
Message-ID: <CABzT7hMKmvNf13p4nedH8hDfpxJ7qdQLG1hZbe6p_GqrHNcEDg@mail.gmail.com>

Ok now I have restructured the matrix. Now I will not use mumps and use
only Petsc and Slepc. I am getting some error.

I will open another thread.

Venkatesh

On Mon, May 25, 2015 at 2:36 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, May 25, 2015 at 2:54 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Ok this will load the matrices in parallel correct ?
>>
>
> Yes
>
>    Matt
>
>
>> On Sun, May 24, 2015 at 7:36 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sun, May 24, 2015 at 8:57 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> I am using Matload option as in the ex7.c code given by the Slepc.
>>>> ierr = MatLoad(A,viewer);CHKERRQ(ierr);
>>>>
>>>>
>>>> There is no problem here right ? or any additional option is required
>>>> for very large matrices while running the eigensolver in parallel ?
>>>>
>>>
>>> This will load the matrix from the viewer (presumably disk). There are
>>> no options for large matrices.
>>>
>>>   Thanks,
>>>
>>>       Matt
>>>
>>>
>>>> cheers,
>>>> Venkatesh
>>>>
>>>> On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Thanks.
>>>>>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>>>>>> submitted in 10 nodes.
>>>>>>
>>>>>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>>>>>
>>>>>
>>>>> The error message from MUMPS said that it tried to allocate 10G. We
>>>>> must assume each process
>>>>> tried to do the same thing. That means if you scheduled 24 processes
>>>>> on a node, it would try to
>>>>> allocate at least 240G, which is in excess of what you specify above.
>>>>>
>>>>> Note that this has nothing to do with PETSc. It is all in the
>>>>> documentation for that machine and its
>>>>> scheduling policy.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Venkatesh
>>>>>>
>>>>>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <
>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>>>>>> Matlab.
>>>>>>>>
>>>>>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in
>>>>>>>> right ?
>>>>>>>>
>>>>>>>
>>>>>>> I do not know how MUMPS allocates memory, but the message is
>>>>>>> unambiguous. Also,
>>>>>>> this is concerned with the memory available per node. Do you know
>>>>>>> how many processes
>>>>>>> per node were scheduled? The message below indicates that it was
>>>>>>> trying to allocate 10G
>>>>>>> for one process.
>>>>>>>
>>>>>>>
>>>>>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>>>>>> running faster.
>>>>>>>>
>>>>>>>
>>>>>>> Yes, for strong scaling you always get slowdown eventually since
>>>>>>> overheads dominate
>>>>>>> work, see Amdahl's Law.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> I have attached the submission script.
>>>>>>>>
>>>>>>>> Pls see.. Kindly let me know
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>> Venkatesh
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <
>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi again,
>>>>>>>>>>
>>>>>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>>>>>> with Mumps.
>>>>>>>>>>
>>>>>>>>>> I am getting this error when I solve eigenvalue problem with
>>>>>>>>>> large matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It ran out of memory on the node.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Also it is again not scaling well for small matrices.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> MUMPS strong scaling for small matrices is not very good. Weak
>>>>>>>>> scaling is looking at big matrices.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Kindly let me know what to do.
>>>>>>>>>>
>>>>>>>>>> cheers,
>>>>>>>>>>
>>>>>>>>>> Venkatesh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I have attached the log of the command which I gave in the
>>>>>>>>>>>> master node: make streams NPMAX=32
>>>>>>>>>>>>
>>>>>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If you look at the STREAMS numbers, you can see that your system
>>>>>>>>>>> is only able to support about 2 cores with the
>>>>>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>>>>>
>>>>>>>>>>> Other codes may do well on this machine, but they would be
>>>>>>>>>>> compute constrained, using things like DGEMM.
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Kindly let me know.
>>>>>>>>>>>>
>>>>>>>>>>>> Venkatesh
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <
>>>>>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Run the streams benchmark on this system and send the
>>>>>>>>>>>>> results.
>>>>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other
>>>>>>>>>>>>> codes scale well.
>>>>>>>>>>>>> > I wanted to ask if this job takes much time, then if I
>>>>>>>>>>>>> submit on more cores, I have to increase the icntl(14).. which would again
>>>>>>>>>>>>> take long time.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > So is there another way ?
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > cheers,
>>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on
>>>>>>>>>>>>> different processors. I had to increase the workspace icntl(14) when I
>>>>>>>>>>>>> submit on more cores since it is failing with small value of icntl(14).
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but
>>>>>>>>>>>>> that is a relatively small effect
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>>>>>> solve. You might try emailing
>>>>>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>>>>>> > and not much more.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >   Thanks,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >      Matt
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>>> > Hi, Thanks for the information. I now increased the
>>>>>>>>>>>>> workspace by adding '-mat_mumps_icntl_14 100'
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I
>>>>>>>>>>>>> get the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100'
>>>>>>>>>>>>> it takes 3500secs.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >     Matt
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2
>>>>>>>>>>>>> -eps_nev 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly
>>>>>>>>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>>>>>>>>> 100'
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Kindly let me know.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Venkatesh
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I run with this command :
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type
>>>>>>>>>>>>> sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>>>>>> processors)
>>>>>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>>>>> ------------------------------------
>>>>>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the
>>>>>>>>>>>>> MUMPS manual. In this case you have INFO(1)=-9, which is explained in the
>>>>>>>>>>>>> manual as:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > "?9 Main internal real/complex workarray S too small. If
>>>>>>>>>>>>> INFO(2) is positive, then the number of entries that are missing in S at
>>>>>>>>>>>>> the moment when the error is raised is available in INFO(2). If INFO(2) is
>>>>>>>>>>>>> negative, then its absolute value should be multiplied by 1 million. If an
>>>>>>>>>>>>> error ?9 occurs, the user should increase the value of ICNTL(14) before
>>>>>>>>>>>>> calling the factorization (JOB=2) again, except if ICNTL(23) is provided,
>>>>>>>>>>>>> in which case ICNTL(23) should be increased."
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > This says that you should use ICTNL(14) to increase the
>>>>>>>>>>>>> working space size:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis
>>>>>>>>>>>>> and the factorization phases. It corresponds to the percentage increase in
>>>>>>>>>>>>> the estimated working space. When significant extra fill-in is caused by
>>>>>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>>>>>> default 20%.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > David
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > --
>>>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > --
>>>>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150525/853842d3/attachment-0001.html>

From venkateshgk.j at gmail.com  Mon May 25 08:18:33 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Mon, 25 May 2015 18:48:33 +0530
Subject: [petsc-users] Memory per core - parallel eigenvalue problem
Message-ID: <CABzT7hMQGX28ujTqxOL+deDCtFU5fru7PuYg5pun=KXxX4fxAQ@mail.gmail.com>

Hi,

I am solving the generalized eigen problem Ax=lambda Bx given in EX7 in
Slepc manual.

Now I have restructured my matrix and I get reasonable answers.

However, when I submit the problem in multiple cores, it is accessing a
large amount of memory than a serial job.

1 Serial job  - Amount of memory used = 18 GB

Parallel job in 240 cores - Amount of memory used = 456 GB

I have attached the performance log for the run.

I think it is loading the full matrix in each core for a parallel job, but
I am not sure how to resolve or whether I am correct..

Kindly help.

cheers,
Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150525/6f9473dd/attachment.html>
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
 Number of iterations of the method: 10
 Number of linear iterations of the method: 99
 Solution method: krylovschur

 Number of requested eigenvalues: 3
 Stopping condition: tol=1e-08, maxit=1600
 Number of converged approximate eigenpairs: 9

           k          ||Ax-kBx||/||kx||
   ----------------- ------------------
 -0.009606+0.000000 i   0.00350922
 -0.009633+0.000000 i    0.0107094
 -0.010251-0.000000 i    0.0104596
 -0.010478+0.000000 i   0.00103213
 -0.011009-0.000000 i  0.000354202
 -0.013199-0.000000 i   0.00022781
 -0.017263+0.000000 i  5.39603e-05
 -0.023652-0.000000 i  2.79115e-05
 -0.032886+0.000000 i  3.21621e-05

************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex7 on a linux-intel named nid01080 with 240 processors, by esdveng Mon May 25 07:39:49 2015
Using Petsc Release Version 3.5.3, Jan, 31, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           1.837e+03      1.00001   1.837e+03
Objects:              1.070e+02      1.00000   1.070e+02
Flops:                6.655e+11      1.00014   6.654e+11  1.597e+14
Flops/sec:            3.622e+08      1.00014   3.621e+08  8.691e+10
Memory:               1.927e+09      1.00475              4.613e+11
MPI Messages:         3.200e+04      1.30610   2.617e+04  6.281e+06
MPI Message Lengths:  2.558e+08     11.04369   1.012e+03  6.357e+09
MPI Reductions:       2.876e+03      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 1.8374e+03 100.0%  1.5969e+14 100.0%  6.281e+06 100.0%  1.012e+03      100.0%  2.875e+03 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

ThreadCommRunKer       1 1.0 1.0970e-03 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
ThreadCommBarrie       1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMult              117 1.0 2.2459e-01 4.5 9.08e+07519.3 4.3e+05 9.2e+02 0.0e+00  0  0  7  6  0   0  0  7  6  0 25951
MatSolve              99 1.0 1.0297e+02 1.1 3.91e+10 1.0 0.0e+00 0.0e+00 0.0e+00  5  6  0  0  0   5  6  0  0  0 91120
MatLUFactorSym         1 1.0 1.0519e+01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatLUFactorNum         1 1.0 1.7081e+03 1.0 6.26e+11 1.0 0.0e+00 0.0e+00 0.0e+00 91 94  0  0  0  91 94  0  0  0 87994
MatConvert             2 1.0 6.9909e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin       4 1.0 3.7676e+0031.4 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         4 1.0 7.5902e-01 1.0 0.00e+00 0.0 5.9e+04 1.2e+02 4.2e+01  0  0  1  0  1   0  0  1  0  1     0
MatGetRow            120 1.0 4.4448e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 6.6685e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetSubMatrice       1 1.0 1.6402e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.6562e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLoad                2 1.0 6.0908e+00 1.0 0.00e+00 0.0 3.2e+04 1.5e+04 4.6e+01  0  0  1  7  2   0  0  1  7  2     0
MatAXPY                1 1.0 2.2365e-01 1.0 0.00e+00 0.0 2.9e+04 1.2e+02 2.2e+01  0  0  0  0  1   0  0  0  0  1     0
MatGetRedundant        1 1.0 2.3025e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMPIConcateSeq       1 1.0 6.6215e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNorm               18 1.0 2.5502e-02 1.1 8.64e+03 1.0 0.0e+00 0.0e+00 1.8e+01  0  0  0  0  1   0  0  0  0  1    81
VecCopy               20 1.0 1.6141e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               213 1.0 2.8761e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                9 1.0 3.1037e-02 1.1 4.32e+03 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    33
VecScatterBegin      315 1.0 1.4699e-01 1.3 0.00e+00 0.0 6.1e+06 9.6e+02 0.0e+00  0  0 97 92  0   0  0 97 92  0     0
VecScatterEnd        315 1.0 7.7702e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
EPSSetUp               1 1.0 1.7233e+03 1.0 6.26e+11 1.0 1.4e+05 3.1e+02 7.9e+01 92 94  2  1  3  92 94  2  1  3 87218
EPSSolve               1 1.0 1.8280e+03 1.0 6.65e+11 1.0 6.1e+06 9.4e+02 2.7e+03 99100 97 91 93  99100 97 91 93 87358
STSetUp                1 1.0 1.7233e+03 1.0 6.26e+11 1.0 1.4e+05 3.1e+02 5.8e+01 92 94  2  1  2  92 94  2  1  2 87218
STApply               99 1.0 1.0423e+02 1.1 3.92e+10 1.0 6.0e+06 9.6e+02 2.0e+02  6  6 95 90  7   6  6 95 90  7 90059
STMatSolve            99 1.0 1.0411e+02 1.1 3.91e+10 1.0 5.7e+06 9.6e+02 2.0e+02  6  6 90 86  7   6  6 90 86  7 90121
BVCopy                19 1.0 2.6266e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01  0  0  0  0  2   0  0  0  0  2     0
BVMult               198 1.0 3.0988e-02 1.0 2.01e+06 1.0 0.0e+00 0.0e+00 3.7e+02  0  0  0  0 13   0  0  0  0 13 15584
BVDot                187 1.0 5.9229e-02 1.0 1.29e+06 1.0 0.0e+00 0.0e+00 1.9e+02  0  0  0  0  7   0  0  0  0  7  5209
BVOrthogonalize      100 1.0 1.3423e-01 1.0 2.48e+06 1.0 0.0e+00 0.0e+00 1.5e+03  0  0  0  0 52   0  0  0  0 52  4436
BVScale              100 1.0 6.8860e-03 1.1 2.40e+04 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   836
BVSetRandom            1 1.0 4.3875e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSSolve               10 1.0 2.4658e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSVectors             20 1.0 3.8042e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSOther               10 1.0 1.1059e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               2 1.0 2.4080e-05 5.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve              99 1.0 1.0409e+02 1.1 3.91e+10 1.0 5.7e+06 9.6e+02 2.0e+02  6  6 90 86  7   6  6 90 86  7 90140
PCSetUp                1 1.0 1.7230e+03 1.0 6.26e+11 1.0 1.2e+05 3.6e+02 2.8e+01 92 94  2  1  1  92 94  2  1  1 87232
PCApply               99 1.0 1.0400e+02 1.1 3.91e+10 1.0 5.7e+06 9.6e+02 0.0e+00  6  6 90 86  0   6  6 90 86  0 90218
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     3              2         1504     0
              Matrix    17             17   1915794800     0
              Vector    56             56       913984     0
      Vector Scatter     6              6         7320     0
           Index Set    15             15       254960     0
Eigenvalue Problem Solver     1              1         2156     0
         PetscRandom     1              1          648     0
  Spectral Transform     1              1          840     0
       Basis Vectors     1              1        10744     0
              Region     1              1          648     0
       Direct solver     1              1        25272     0
       Krylov Solver     2              2         2320     0
      Preconditioner     2              2         1928     0
========================================================================================================================
Average time to get PetscTime(): 0
Average time for MPI_Barrier(): 1.54018e-05
Average time for zero size MPI_Send(): 4.47432e-06
#PETSc Option Table entries:
-eps_nev 3
-eps_target 0.5
-f1 a40t
-f2 b40t
-log_summary
-st_type sinvert
#End of PETSc Option Table entries
Compiled with FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 sizeof(PetscInt) 4
Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/home/proj/14/esdveng/apps/valgrind-3.10.1 --download-mumps=/home/proj/14/esdveng/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz
-----------------------------------------
Libraries compiled on Tue May 19 06:39:06 2015 on login4 
Machine characteristics: Linux-3.0.101-0.31.1_1.0502.8394-cray_ari_s-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/lustre/esd2/esdveng/petsc-3.5.3
Using PETSc arch: linux-intel
-----------------------------------------

Using C compiler: gcc  -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ifort  -fPIC -g   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/include -I/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/include
-----------------------------------------

Using C linker: gcc
Using Fortran linker: ifort
Using libraries: -Wl,-rpath,/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -L/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -lpetsc -Wl,-rpath,/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -L/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/lib -L/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/lib -lmpich -lssl -luuid -lpthread -lrt -ldl -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lpthread -lirc_s -lm -lstdc++ -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -lstdc++ -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -ldl -lgcc_s -ldl  
-----------------------------------------

Application 404794 resources: utime ~438984s, stime ~1036s, Rss ~1905660, inblocks ~1047499, outblocks ~33037

From jroman at dsic.upv.es  Mon May 25 08:26:16 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Mon, 25 May 2015 15:26:16 +0200
Subject: [petsc-users] Memory per core - parallel eigenvalue problem
In-Reply-To: <CABzT7hMQGX28ujTqxOL+deDCtFU5fru7PuYg5pun=KXxX4fxAQ@mail.gmail.com>
References: <CABzT7hMQGX28ujTqxOL+deDCtFU5fru7PuYg5pun=KXxX4fxAQ@mail.gmail.com>
Message-ID: <241F4503-3C4D-4BD1-AAE7-B1B8263E4821@dsic.upv.es>

Once again, you need a parallel linear solver.
Read FAQ #6 and the corresponding section on the users manual.
http://slepc.upv.es/documentation/faq.htm
Jose


El 25/05/2015, a las 15:18, venkatesh g escribi?:

> Hi, 
> 
> I am solving the generalized eigen problem Ax=lambda Bx given in EX7 in Slepc manual.
> 
> Now I have restructured my matrix and I get reasonable answers.
> 
> However, when I submit the problem in multiple cores, it is accessing a large amount of memory than a serial job. 
> 
> 1 Serial job  - Amount of memory used = 18 GB
> 
> Parallel job in 240 cores - Amount of memory used = 456 GB
> 
> I have attached the performance log for the run. 
> 
> I think it is loading the full matrix in each core for a parallel job, but I am not sure how to resolve or whether I am correct.. 
> 
> Kindly help.
> 
> cheers,
> Venkatesh
> 
> <performance_log.txt>


From daaugusto at gmail.com  Mon May 25 13:39:54 2015
From: daaugusto at gmail.com (Douglas A. Augusto)
Date: Mon, 25 May 2015 15:39:54 -0300
Subject: [petsc-users] Current state of OpenMP support in PETSc
In-Reply-To: <87pp5sv2q9.fsf@jedbrown.org>
References: <20150522145001.GA1163@localhost.localdomain>
	<87pp5sv2q9.fsf@jedbrown.org>
Message-ID: <55636c6f.72258c0a.7d86.ffffb6d0@mx.google.com>

On 22/05/2015 at 13:20,
Jed Brown <jed at jedbrown.org> wrote:

> We're also developing shared memory MPI-based approaches and would
> recommend that you plan to not use OpenMP (it's a horrible programming
> model -- oversynchronizing and without a concept of memory locality).

Thanks for the clarification. Does PETSc already take advantage of any of these
MPI-based shared memory approaches?


Best regards,

-- 
Douglas A. Augusto

From jed at jedbrown.org  Mon May 25 18:04:14 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 26 May 2015 01:04:14 +0200
Subject: [petsc-users] Current state of OpenMP support in PETSc
In-Reply-To: <55636c6f.72258c0a.7d86.ffffb6d0@mx.google.com>
References: <20150522145001.GA1163@localhost.localdomain>
	<87pp5sv2q9.fsf@jedbrown.org>
	<55636c6f.72258c0a.7d86.ffffb6d0@mx.google.com>
Message-ID: <87twv0b6oh.fsf@jedbrown.org>

"Douglas A. Augusto" <daaugusto at gmail.com> writes:
> Thanks for the clarification. Does PETSc already take advantage of any of these
> MPI-based shared memory approaches?

No.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/bd8de88b/attachment.pgp>

From vijay.gopal.c at gmail.com  Tue May 26 00:41:04 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Tue, 26 May 2015 07:41:04 +0200
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
In-Reply-To: <C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>
References: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
	<C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>
Message-ID: <CAJa9Fi_zpC_7Q-_9H5EMzsqjifMkSw5NaDVHyowJr_eOimz=CQ@mail.gmail.com>

Thanks Barry,

You were right ! when i removed the COPTFLAGS and recompiled the same code
worked.
It now seems to be working perfectly.

So somehow the optimization flags effect the code execution !?
is this a known issue ?


thanks anyways,
 Vijay

On Sat, May 23, 2015 at 9:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    This error message is almost always because a Nan or Inf has crept into
> the numbers and then gotten into a norm or inner product.
> You can run with -fp_trap or run a debugger and have it catch floating
> point exceptions to see the first point where the Nan or Inf appeared and
> help track down the cause.  But do all this tracking down using a debug
> version of the library without all the extra optimization flags you have
> put into the COPTFLAGS flags etc.
>
>   Barry
>
> > On May 23, 2015, at 12:27 PM, Vijay Gopal Chilkuri <
> vijay.gopal.c at gmail.com> wrote:
> >
> > Hi,
> >
> > I'm having problems calculating large number of eigenvalues(nev>400)
> with SLEPc, i get perfect results with small number of eigenvalues.
> >
> >
> > PETSc: Branch: origin/maint ; commit: 2b04bc0
> > SLEPc: Branch: origin/maint ; commit: e1f03d9
> >
> > This is the error message i get:
> >
> > Intel Parallel Studio XE 2013 loaded
> > Intel(R) MPI Library 4.1 (4.1.3.049) loaded
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Invalid argument
> > [0]PETSC ERROR: Scalar value must be same on all processes, argument # 3
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown?
> > [0]PETSC ERROR: ./ex1 on a arch-linux2-c-opt named eoscomp7 by vijayc
> Sat May 23 19:04:55 2015
> > [0]PETSC ERROR: Configure options --with-64-bit-indices --with-cc=mpiicc
> --with-fc=mpiifort --with-cxx=mpiicpc --with-debugging=1 --FOPTFLAGS="-O3
> -xAV
> > X -fno-alias -no-prec-div -no-prec-sqrt -ip " --COPTFLAGS="-O3 -xAVX
> -fno-alias -no-prec-div -no-prec-sqrt -ip " --CXXOPTFLAGS="-O3 -xAVX
> -fno-alias -no-prec-div -no-prec-sqrt -ip " --download-fblaslapack
> --with-x=false
> > [0]PETSC ERROR: #1 BVScaleColumn() line 380 in
> /eos1/p1517/vijayc/slepc_basic/src/sys/classes/bv/interface/bvops.c
> > [0]PETSC ERROR: #2 EPSFullLanczos() line 200 in
> /eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylov.c
> > [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Symm() line 56 in
> /eos1/p1517/vijayc/slepc_basic/src/eps/impls/krylov/krylovschur/ks-symm.c
> > [0]PETSC ERROR: #4 EPSSolve() line 99 in
> /eos1/p1517/vijayc/slepc_basic/src/eps/interface/epssolve.c
> > [0]PETSC ERROR: #5 main() line 143 in
> /users/p1517/vijayc/modelization/ntrou3/sites_18/obc_10/isz_0/ex1.c
> > [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> > application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
> >
> > thanks a lot,
> >  Vijay
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/8af028d3/attachment.html>

From venkateshgk.j at gmail.com  Tue May 26 01:46:11 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Tue, 26 May 2015 12:16:11 +0530
Subject: [petsc-users] MUMPS issue
Message-ID: <CABzT7hMF1mJ18HWS4psjNxT_Rd9K+auPyLe2TtCbaC_Ad2BwJg@mail.gmail.com>

Thanks. I now have a better matrix B in my eigenproblem than last time. Now
I included MUMPS. I get this error when I give large matrices to solve:

'Error reported by MUMPS in numerical factorization phase: INFO(1)=-1,
INFO(2)=0'

I checked the manual, there is only one instance of INFO(2)=0 and I am not
able to get it.

Can someone help.

cheers,

Venkatesh



On Mon, May 25, 2015 at 6:56 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:

> Once again, you need a parallel linear solver.
> Read FAQ #6 and the corresponding section on the users manual.
> http://slepc.upv.es/documentation/faq.htm
> Jose
>
>
> El 25/05/2015, a las 15:18, venkatesh g escribi?:
>
> > Hi,
> >
> > I am solving the generalized eigen problem Ax=lambda Bx given in EX7 in
> Slepc manual.
> >
> > Now I have restructured my matrix and I get reasonable answers.
> >
> > However, when I submit the problem in multiple cores, it is accessing a
> large amount of memory than a serial job.
> >
> > 1 Serial job  - Amount of memory used = 18 GB
> >
> > Parallel job in 240 cores - Amount of memory used = 456 GB
> >
> > I have attached the performance log for the run.
> >
> > I think it is loading the full matrix in each core for a parallel job,
> but I am not sure how to resolve or whether I am correct..
> >
> > Kindly help.
> >
> > cheers,
> > Venkatesh
> >
> > <performance_log.txt>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/f2df5eeb/attachment.html>

From jed at jedbrown.org  Tue May 26 02:14:33 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 26 May 2015 09:14:33 +0200
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
In-Reply-To: <CAJa9Fi_zpC_7Q-_9H5EMzsqjifMkSw5NaDVHyowJr_eOimz=CQ@mail.gmail.com>
References: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
	<C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>
	<CAJa9Fi_zpC_7Q-_9H5EMzsqjifMkSw5NaDVHyowJr_eOimz=CQ@mail.gmail.com>
Message-ID: <87lhgbbyjq.fsf@jedbrown.org>

Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> writes:
> So somehow the optimization flags effect the code execution !?
> is this a known issue ?

Either your code is invalid (e.g., undefined behavior), numerically
unstable, or your vendor has a bug that you should report.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/44a89fe6/attachment.pgp>

From Manishkumar.K at LntTechservices.com  Tue May 26 06:42:04 2015
From: Manishkumar.K at LntTechservices.com (Manish Kumar K)
Date: Tue, 26 May 2015 11:42:04 +0000
Subject: [petsc-users] Reg :PETSc  Build error with mpi option
Message-ID: <AA7F30F772C09F47AE19F45AD9B68B785515CC94@POCITMSEXMB06.LntUniverse.com>

Hi PETSc Team,

                  Greetings of the day!

I am facing error while building building PETSc using mpich option

Please let me know where am I going wrong .


20063086 at BLRMIESPC-745 /package/petsc-3.5.3

$ ./configure --with-cc=gcc --with-cxx=g++ --download-mpich="D:/Software/cygwin/package/extpack/mpich-3.1.4" --download-fblaslapack="D:/Software/cygwin/package/                                            extpack/fblaslapack.tar.gz" --with-batch
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
      Running configure on MPICH; this may take several minutes
   ===============================================================================                                                                                                                                                                         *******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Error running configure on MPICH.
   On Microsoft Windows systems, please obtain and install the binary distribution from
     http://www.mcs.anl.gov/mpi/mpich/mpich-nt
   then rerun configure.
   If you choose to install MPICH to a location other than the default, use
     --with-mpi-dir=<directory>
   to specify the location of the installation when you rerun configure.
*******************************************************************************


The following command also gives me same error

$ ./configure --with-cc=gcc --with-cxx=g++ --with-mpich=/cygdrive/D/Software/cygwin/package/extpack/mpich-3.1.4.tar.gz  --with-fblaslapack=/cygdrive/D/Software/cygwin/package/                                            extpack/fblaslapack.tar.gz --with-batch

Help me out rectifying my build issue.


Thanks & Regards
Manish K



L&T Technology Services Ltd

www.LntTechservices.com<http://www.lnttechservices.com/>

This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/7fa4c5fb/attachment-0001.html>

From dave.mayhem23 at gmail.com  Tue May 26 06:58:14 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 26 May 2015 13:58:14 +0200
Subject: [petsc-users] Reg :PETSc Build error with mpi option
In-Reply-To: <AA7F30F772C09F47AE19F45AD9B68B785515CC94@POCITMSEXMB06.LntUniverse.com>
References: <AA7F30F772C09F47AE19F45AD9B68B785515CC94@POCITMSEXMB06.LntUniverse.com>
Message-ID: <CAJ98EDrg22dSsTVjfEDZLgctaoFrCyJQcXeGPD0WV_DvLL=8Eg@mail.gmail.com>

You should send the entire configure.log file to petsc-maint at mcs.anl.gov so
people can help you.

On 26 May 2015 at 13:42, Manish Kumar K <Manishkumar.K at lnttechservices.com>
wrote:

>  Hi PETSc Team,
>
>
>
>                   Greetings of the day!
>
>
>
> I am facing error while building building PETSc using mpich option
>
>
>
> Please let me know where am I going wrong .
>
>
>
>
>
> 20063086 at BLRMIESPC-745 /package/petsc-3.5.3
>
>
>
> $ ./configure --with-cc=gcc --with-cxx=g++
> --download-mpich="D:/Software/cygwin/package/extpack/mpich-3.1.4"
> --download-fblaslapack="D:/Software/cygwin/package/
> extpack/fblaslapack.tar.gz" --with-batch
>
>
> ===============================================================================
>
>              Configuring PETSc to compile on your system
>
>
> ===============================================================================
>
>       Running configure on MPICH; this may take several
> minutes
>
>    ===============================================================================
> *******************************************************************************
>
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
> details):
>
>
> -------------------------------------------------------------------------------
>
> Error running configure on MPICH.
>
>    On Microsoft Windows systems, please obtain and install the binary
> distribution from
>
>      http://www.mcs.anl.gov/mpi/mpich/mpich-nt
>
>    then rerun configure.
>
>    If you choose to install MPICH to a location other than the default, use
>
>      --with-mpi-dir=<directory>
>
>    to specify the location of the installation when you rerun configure.
>
>
> *******************************************************************************
>
>
>
>
>
> The following command also gives me same error
>
>
>
> $ ./configure --with-cc=gcc --with-cxx=g++
> --with-mpich=/cygdrive/D/Software/cygwin/package/extpack/mpich-3.1.4.tar.gz
>  --with-fblaslapack=/cygdrive/D/Software/cygwin/package/
> extpack/fblaslapack.tar.gz --with-batch
>
>
>
> Help me out rectifying my build issue.
>
>
>
>
>
> Thanks & Regards
>
> Manish K
>
>
>
>
>
> *L&T Technology Services Ltd*
>
> www.LntTechservices.com <http://www.lnttechservices.com/>
>
> This Email may contain confidential or privileged information for the
> intended recipient (s). If you are not the intended recipient, please do
> not use or disseminate the information, notify the sender and delete it
> from your system.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/0177575c/attachment.html>

From venkateshgk.j at gmail.com  Tue May 26 07:01:29 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Tue, 26 May 2015 17:31:29 +0530
Subject: [petsc-users] Pastix with Petsc and Slepc
Message-ID: <CABzT7hOCQtiQxXfcTK7euA=cTaTPcZrcGzQmnU_Ey6vk_23iww@mail.gmail.com>

Hi I am trying to solve my eigenproblem Ax=lambdaBx with a direct linear
solver Pastix

However I get this error (attached). It works without the pastix option.

My matrices are in binary format.

I gave the command : aprun -n 24 -N 24 ./ex7 -f1 a20t -f2 b20t -st_type
sinvert -eps_nev 3 -log_summary -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_package pastix -ksp_monitor

Kindly help me out.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/6c43b10c/attachment-0001.html>
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...

ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: 
ERROR: The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct format



The matrix is not in the correct format





The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format







The matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct format

The matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct formatThe matrix is not in the correct format









The matrix is not in the correct format







The matrix is not in the correct format

[9]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[15]PETSC ERROR: [14]PETSC ERROR: [4]PETSC ERROR: ------------------------------------------------------------------------
------------------------------------------------------------------------
[15]PETSC ERROR: [9]PETSC ERROR: [14]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[5]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: [4]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[15]PETSC ERROR: [14]PETSC ERROR: [1]PETSC ERROR: [13]PETSC ERROR: [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[5]PETSC ERROR: [13]PETSC ERROR: [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
------------------------------------------------------------------------
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: ------------------------------------------------------------------------
------------------------------------------------------------------------
------------------------------------------------------------------------
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: [21]PETSC ERROR: [19]PETSC ERROR: [18]PETSC ERROR: [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[13]PETSC ERROR: [21]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: [19]PETSC ERROR: [21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
Try option -start_in_debugger or -on_error_attach_debugger
Try option -start_in_debugger or -on_error_attach_debugger
[21]PETSC ERROR: [13]PETSC ERROR: [19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[18]PETSC ERROR: [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
likely location of problem given in stack below
[15]PETSC ERROR: [14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[14]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------
Note: The EXACT line numbers in the stack are not available,
---------------------  Stack Frames ------------------------------------
[7]PETSC ERROR: [21]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
      INSTEAD the line number of the start of the function
Note: The EXACT line numbers in the stack are not available,
[7]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: likely location of problem given in stack below
      is given.
      INSTEAD the line number of the start of the function
[14]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      is given.
---------------------  Stack Frames ------------------------------------
[8]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [13]PETSC ERROR: [15] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[14] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[15]PETSC ERROR: likely location of problem given in stack below
Note: The EXACT line numbers in the stack are not available,
[18]PETSC ERROR: [19]PETSC ERROR: [13]PETSC ERROR: [8]PETSC ERROR: [9]PETSC ERROR: [15] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[14]PETSC ERROR: likely location of problem given in stack below
likely location of problem given in stack below
      INSTEAD the line number of the start of the function
---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[19]PETSC ERROR: [18]PETSC ERROR: [13]PETSC ERROR: [15]PETSC ERROR: [14] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[15] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[9]PETSC ERROR: [8]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
---------------------  Stack Frames ------------------------------------
[15]PETSC ERROR: [14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
[15] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[9]PETSC ERROR: [8]PETSC ERROR: [15]PETSC ERROR: [14] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: [15] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
      INSTEAD the line number of the start of the function
Note: The EXACT line numbers in the stack are not available,
Note: The EXACT line numbers in the stack are not available,
Note: The EXACT line numbers in the stack are not available,
[15]PETSC ERROR: [14] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[18]PETSC ERROR: [19]PETSC ERROR: [14]PETSC ERROR: [15] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[9]PETSC ERROR: [3]PETSC ERROR: [10]PETSC ERROR: [12]PETSC ERROR: [14] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
      INSTEAD the line number of the start of the function
[8]PETSC ERROR: [22]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [9]PETSC ERROR:       is given.
      is given.
[15] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[14] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[8]PETSC ERROR: [9]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [8] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[9] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      INSTEAD the line number of the start of the function
      INSTEAD the line number of the start of the function
------------------------------------------------------------------------
[20]PETSC ERROR: [9]PETSC ERROR: [18]PETSC ERROR: [19]PETSC ERROR:       is given.
[15] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[14] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      is given.
[18]PETSC ERROR: [3]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: [14]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[18] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[19]PETSC ERROR: [14] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[9] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[19] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[18]PETSC ERROR: [3]PETSC ERROR: [18] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[19]PETSC ERROR: [9]PETSC ERROR: [20]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: [19] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[9] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[14] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[15] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[9]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[14] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[15] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[9] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[3]PETSC ERROR: [19]PETSC ERROR: [18] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[18]PETSC ERROR: [3]PETSC ERROR: [9]PETSC ERROR: [18] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[19]PETSC ERROR: [20]PETSC ERROR: [9] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Try option -start_in_debugger or -on_error_attach_debugger
[19] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[18]PETSC ERROR: [19]PETSC ERROR: [9]PETSC ERROR: [19] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[18] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[9] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [19]PETSC ERROR: [18]PETSC ERROR: [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[19] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[18] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: [9] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[19]PETSC ERROR: [18]PETSC ERROR: [9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[18] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[15]PETSC ERROR: [14]PETSC ERROR: [9] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[18]PETSC ERROR: [19] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
Signal received
Signal received
[19]PETSC ERROR: [18] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[9]PETSC ERROR: ------------------------------------------------------------------------
[9] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[14]PETSC ERROR: [15]PETSC ERROR: [18]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: [6]PETSC ERROR: [1]PETSC ERROR: [9] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
likely location of problem given in stack below
[14]PETSC ERROR: [15]PETSC ERROR: [9]PETSC ERROR: [20]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[9] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[3]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: likely location of problem given in stack below
---------------------  Stack Frames ------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[3]PETSC ERROR: [20]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[14]PETSC ERROR: [20]PETSC ERROR: [15]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[9]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Note: The EXACT line numbers in the stack are not available,
--------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: [5]PETSC ERROR:       INSTEAD the line number of the start of the function
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Signal received
[20]PETSC ERROR: [5]PETSC ERROR: [9]PETSC ERROR:       is given.
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: [5]PETSC ERROR: [9]PETSC ERROR: [20] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Petsc Release Version 3.5.3, Jan, 31, 2015 
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[20]PETSC ERROR: [9]PETSC ERROR: [4]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[20] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: [20]PETSC ERROR: [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[2]PETSC ERROR: [9]PETSC ERROR: [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
#1 User provided function() line 0 in  unknown file
[2]PETSC ERROR: likely location of problem given in stack below
Try option -start_in_debugger or -on_error_attach_debugger
      is given.
[2]PETSC ERROR: [21]PETSC ERROR: [13]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[13] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[5]PETSC ERROR: [21]PETSC ERROR: [2]PETSC ERROR: likely location of problem given in stack below
Note: The EXACT line numbers in the stack are not available,
Rank 14 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14
Rank 15 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15
[5]PETSC ERROR: [13]PETSC ERROR: [16]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: [13] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[7]PETSC ERROR: [4]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
likely location of problem given in stack below
[13]PETSC ERROR: likely location of problem given in stack below
      INSTEAD the line number of the start of the function
[5]PETSC ERROR: [4]PETSC ERROR: [21]PETSC ERROR: [7]PETSC ERROR: ------------------------------------------------------------------------
---------------------  Stack Frames ------------------------------------
---------------------  Stack Frames ------------------------------------
      is given.
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[7]PETSC ERROR: Rank 9 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9
[13] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Note: The EXACT line numbers in the stack are not available,
[17]PETSC ERROR: [16]PETSC ERROR: [7]PETSC ERROR: [13]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR: [13] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
      INSTEAD the line number of the start of the function
[13]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR: ------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[7]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [2]PETSC ERROR:       INSTEAD the line number of the start of the function
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[16]PETSC ERROR: [21]PETSC ERROR: [13] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[21] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
      is given.
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[17]PETSC ERROR: [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[23]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [7]PETSC ERROR: [13] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
[17]PETSC ERROR: [7] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[13]PETSC ERROR: [7] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[23]PETSC ERROR: [13] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[7] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: [23]PETSC ERROR: [7]PETSC ERROR: [13] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[13]PETSC ERROR: [7] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[23]PETSC ERROR: [7]PETSC ERROR: [13] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[13]PETSC ERROR: [7]PETSC ERROR: [13] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[7] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
------------------------------------------------------------------------
[7]PETSC ERROR: [13]PETSC ERROR: [7] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22]PETSC ERROR: [16]PETSC ERROR: [13] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
likely location of problem given in stack below
[7]PETSC ERROR: [22]PETSC ERROR: [7] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
------------------------------------------------------------------------
Try option -start_in_debugger or -on_error_attach_debugger
[16]PETSC ERROR: [17]PETSC ERROR: [22]PETSC ERROR: likely location of problem given in stack below
---------------------  Stack Frames ------------------------------------
[12]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
[22]PETSC ERROR: [23]PETSC ERROR: [16]PETSC ERROR: [17]PETSC ERROR: [12]PETSC ERROR: [13]PETSC ERROR: [7] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
      INSTEAD the line number of the start of the function
--------------------- Error Message --------------------------------------------------------------
Try option -start_in_debugger or -on_error_attach_debugger
likely location of problem given in stack below
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[23]PETSC ERROR: [12]PETSC ERROR: [13]PETSC ERROR: [11]PETSC ERROR: Signal received
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[7]PETSC ERROR: ------------------------------------------------------------------------
[7] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[16]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
      is given.
---------------------  Stack Frames ------------------------------------
[16]PETSC ERROR: [17]PETSC ERROR: ------------------------------------------------------------------------
[23]PETSC ERROR: [10]PETSC ERROR:       INSTEAD the line number of the start of the function
[16] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Note: The EXACT line numbers in the stack are not available,
[17]PETSC ERROR: [16]PETSC ERROR: [23]PETSC ERROR: [10]PETSC ERROR:       is given.
[16] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[17]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[17] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[16]PETSC ERROR: [10]PETSC ERROR: [22]PETSC ERROR: [16] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[16]PETSC ERROR: [17] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[10]PETSC ERROR: [16] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[17]PETSC ERROR: likely location of problem given in stack below
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[22]PETSC ERROR: [16]PETSC ERROR: [17] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
---------------------  Stack Frames ------------------------------------
[16] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[17]PETSC ERROR: [22]PETSC ERROR: [16]PETSC ERROR: [17] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
Note: The EXACT line numbers in the stack are not available,
[16] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [17] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[16] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
      INSTEAD the line number of the start of the function
[17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [16] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[17] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR:       is given.
[16] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[17]PETSC ERROR: [22]PETSC ERROR: [16]PETSC ERROR: [17] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[17]PETSC ERROR: [16] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[17] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[10]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [17] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[22] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
likely location of problem given in stack below
[17]PETSC ERROR: [16] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[17] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[22]PETSC ERROR: [17]PETSC ERROR: [22] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[22]PETSC ERROR: [8]PETSC ERROR: [22] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[22]PETSC ERROR: [8] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[22] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[8]PETSC ERROR: [22]PETSC ERROR: [0]PETSC ERROR: [22] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[8] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: [1]PETSC ERROR: [8]PETSC ERROR: [22] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
Try option -start_in_debugger or -on_error_attach_debugger
[8] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[1]PETSC ERROR: [8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[22]PETSC ERROR: [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[22] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[8] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
--------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: [1]PETSC ERROR: [16]PETSC ERROR: [17]PETSC ERROR: [22] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Signal received
Signal received
[8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[22]PETSC ERROR: [8]PETSC ERROR: [17]PETSC ERROR: [16]PETSC ERROR: [8] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[19] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[8]PETSC ERROR: [22] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[8] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[16]PETSC ERROR: [17]PETSC ERROR: [22]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[8]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[16]PETSC ERROR: [8] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[17]PETSC ERROR: [8]PETSC ERROR: [19]PETSC ERROR: [8] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[17]PETSC ERROR: [19] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
#1 User provided function() line 0 in  unknown file
[16]PETSC ERROR: [19]PETSC ERROR: [1]PETSC ERROR: [19] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
#1 User provided function() line 0 in  unknown file
likely location of problem given in stack below
[19]PETSC ERROR: [1]PETSC ERROR: [19] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
---------------------  Stack Frames ------------------------------------
[18] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: [6]PETSC ERROR: [8]PETSC ERROR: [18]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[18] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Try option -start_in_debugger or -on_error_attach_debugger
--------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: [6]PETSC ERROR: [8]PETSC ERROR: [18] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Signal received
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: [8]PETSC ERROR: [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
      INSTEAD the line number of the start of the function
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[8]PETSC ERROR: [1]PETSC ERROR: [3]PETSC ERROR:       is given.
Petsc Release Version 3.5.3, Jan, 31, 2015 
[19]PETSC ERROR: [1]PETSC ERROR:       INSTEAD the line number of the start of the function
[8]PETSC ERROR: [1] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
--------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[1]PETSC ERROR: [3]PETSC ERROR: [18]PETSC ERROR: [19]PETSC ERROR: [8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
Signal received
      is given.
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[18]PETSC ERROR: [19]PETSC ERROR: [8]PETSC ERROR: Signal received
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#1 User provided function() line 0 in  unknown file
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19]PETSC ERROR: likely location of problem given in stack below
[18]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
---------------------  Stack Frames ------------------------------------
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Note: The EXACT line numbers in the stack are not available,
[19]PETSC ERROR: [18]PETSC ERROR: [6]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
      INSTEAD the line number of the start of the function
[20] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: [18]PETSC ERROR: [20]PETSC ERROR:       is given.
#1 User provided function() line 0 in  unknown file
[20] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[6]PETSC ERROR: [20]PETSC ERROR: [6] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[20] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[6]PETSC ERROR: [20]PETSC ERROR: [6] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Rank 8 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8
[6]PETSC ERROR: [20] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[6] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[20]PETSC ERROR: [6]PETSC ERROR: [20] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[6] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[20]PETSC ERROR: [6]PETSC ERROR: [20] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[6] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[20]PETSC ERROR: likely location of problem given in stack below
[6]PETSC ERROR: [20] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[6] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [2]PETSC ERROR: [20] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[6]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[6] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[20]PETSC ERROR: [6]PETSC ERROR: [2]PETSC ERROR: [6] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[20] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Note: The EXACT line numbers in the stack are not available,
[6]PETSC ERROR:       is given.
[2]PETSC ERROR: [6] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR:       INSTEAD the line number of the start of the function
[4]PETSC ERROR: [6]PETSC ERROR: [2]PETSC ERROR: [6] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
      is given.
[5] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[6]PETSC ERROR:       is given.
[5]PETSC ERROR: [4]PETSC ERROR: [2]PETSC ERROR: [6] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[5] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[20]PETSC ERROR: [2] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[5]PETSC ERROR: [4] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Rank 19 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19
Rank 18 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18
[2]PETSC ERROR: [4]PETSC ERROR: [5] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[4] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[2] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[4]PETSC ERROR: [5]PETSC ERROR: [4] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: [21]PETSC ERROR: Signal received
[4]PETSC ERROR: [5] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[2]PETSC ERROR: [4] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[5]PETSC ERROR: [20]PETSC ERROR: [21] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: [5] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[4]PETSC ERROR: [2] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[4] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[5]PETSC ERROR: [2]PETSC ERROR: [5] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: [21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[4] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: [20]PETSC ERROR: [21] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[2] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[5] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[2]PETSC ERROR: [11]PETSC ERROR: [20]PETSC ERROR: [2] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[4] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[21]PETSC ERROR: [20]PETSC ERROR: [2]PETSC ERROR: [5] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[4]PETSC ERROR: [2] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[11]PETSC ERROR: [2]PETSC ERROR: [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[2] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[21] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[5] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[20]PETSC ERROR: [21]PETSC ERROR: [4] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[5]PETSC ERROR: [4]PETSC ERROR: [11]PETSC ERROR: [4] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
#1 User provided function() line 0 in  unknown file
[2]PETSC ERROR: [21]PETSC ERROR: [5]PETSC ERROR: [4]PETSC ERROR: [21] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[4] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[2] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[4]PETSC ERROR: [2]PETSC ERROR: [21]PETSC ERROR: [4] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[2] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[21] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[2]PETSC ERROR: [21]PETSC ERROR: [2] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[21] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[2]PETSC ERROR: [21]PETSC ERROR: [2] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[11]PETSC ERROR: [21] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
likely location of problem given in stack below
[7]PETSC ERROR: [11]PETSC ERROR: [5]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[7] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[21]PETSC ERROR: [11]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: [4]PETSC ERROR: Signal received
[21] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Rank 20 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20
[11]PETSC ERROR: [5]PETSC ERROR: Signal received
      INSTEAD the line number of the start of the function
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: [11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR:       is given.
[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
Signal received
Petsc Release Version 3.5.3, Jan, 31, 2015 
[2]PETSC ERROR: [11]PETSC ERROR: [5]PETSC ERROR: [4]PETSC ERROR: [7]PETSC ERROR: [11] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Signal received
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Petsc Release Version 3.5.3, Jan, 31, 2015 
[5]PETSC ERROR: [21]PETSC ERROR: [2]PETSC ERROR: [7]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[4]PETSC ERROR: [11] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
--------------------- Error Message --------------------------------------------------------------
Petsc Release Version 3.5.3, Jan, 31, 2015 
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[5]PETSC ERROR: [4]PETSC ERROR: [7]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
#1 User provided function() line 0 in  unknown file
[21]PETSC ERROR: [4]PETSC ERROR: Signal received
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[21]PETSC ERROR: [11]PETSC ERROR: [7]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[11] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: [21]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Petsc Release Version 3.5.3, Jan, 31, 2015 
[11] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[7]PETSC ERROR: [21]PETSC ERROR: [11]PETSC ERROR: #1 User provided function() line 0 in  unknown file
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[11] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[21]PETSC ERROR: [12]PETSC ERROR: [11]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[11] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: [11]PETSC ERROR: [13]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[11] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
      INSTEAD the line number of the start of the function
[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[23]PETSC ERROR: [11] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      is given.
[11]PETSC ERROR: [23]PETSC ERROR: [13]PETSC ERROR: [11] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[11]PETSC ERROR: [23]PETSC ERROR: Rank 4 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4
Rank 5 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5
[13]PETSC ERROR: [23] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Rank 7 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7
[23]PETSC ERROR: [11]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[11] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[23] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: [12]PETSC ERROR: [23]PETSC ERROR: likely location of problem given in stack below
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[12]PETSC ERROR: [23] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
---------------------  Stack Frames ------------------------------------
[13]PETSC ERROR: [12]PETSC ERROR: Rank 21 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21
Note: The EXACT line numbers in the stack are not available,
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [12]PETSC ERROR: ------------------------------------------------------------------------
      INSTEAD the line number of the start of the function
---------------------  Stack Frames ------------------------------------
[12]PETSC ERROR:       is given.
[22] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[10]PETSC ERROR: [11]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: [11]PETSC ERROR: [12] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Signal received
[1] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: [11]PETSC ERROR: [12] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[22]PETSC ERROR: [12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: [12]PETSC ERROR: Signal received
[12] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[22]PETSC ERROR: Rank 17 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17
[12]PETSC ERROR: Rank 13 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[22]PETSC ERROR: Rank 16 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16
[12]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[12] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[22]PETSC ERROR: [12]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[12] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[22]PETSC ERROR: [12]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[12] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[22]PETSC ERROR: [12]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[12] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[12]PETSC ERROR: [12] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[12]PETSC ERROR: [12] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Signal received
[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[12]PETSC ERROR: [3]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
Rank 22 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22
[12]PETSC ERROR: [3] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [12]PETSC ERROR: [6]PETSC ERROR: [3] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
#1 User provided function() line 0 in  unknown file
[3]PETSC ERROR: Signal received
[3] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[6]PETSC ERROR: [3]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[6]PETSC ERROR: [3] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[3]PETSC ERROR: [6]PETSC ERROR: [3] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
#1 User provided function() line 0 in  unknown file
[3]PETSC ERROR: [2]PETSC ERROR: [23]PETSC ERROR: [3] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[3]PETSC ERROR: [23] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[2]PETSC ERROR: [3] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23]PETSC ERROR: [3]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[23] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: [3] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[3]PETSC ERROR: [23]PETSC ERROR: [2]PETSC ERROR: [3] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[23] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
#1 User provided function() line 0 in  unknown file
[23]PETSC ERROR: [0]PETSC ERROR: [23] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
      INSTEAD the line number of the start of the function
[23]PETSC ERROR: [1] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[23] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: [1]PETSC ERROR:       is given.
Rank 6 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6
[23]PETSC ERROR: [1] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[10]PETSC ERROR: [23] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[1]PETSC ERROR: [10] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[23]PETSC ERROR: [1] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[23] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[3]PETSC ERROR: [1]PETSC ERROR: [10]PETSC ERROR: [11]PETSC ERROR: [10] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[1] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
--------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [11]PETSC ERROR: [3]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[1]PETSC ERROR: [10]PETSC ERROR: [11]PETSC ERROR: [1] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[10] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [1]PETSC ERROR: Rank 2 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
[10] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[1] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[23]PETSC ERROR: [10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: [10] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[23]PETSC ERROR: [1] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: Signal received
[1]PETSC ERROR: [23]PETSC ERROR: [10] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[1] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: [23]PETSC ERROR: [1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Rank 12 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 12
[23]PETSC ERROR: [10] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[1] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [23]PETSC ERROR: [10] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10]PETSC ERROR: [23]PETSC ERROR: [10] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
#1 User provided function() line 0 in  unknown file
[10]PETSC ERROR: [10] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
Rank 11 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Signal received
[1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
Signal received
[1]PETSC ERROR: [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[10]PETSC ERROR: [3]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[10]PETSC ERROR: [1]PETSC ERROR: [3]PETSC ERROR: Signal received
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[10]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: [3]PETSC ERROR: [10]PETSC ERROR: #1 User provided function() line 0 in  unknown file
#1 User provided function() line 0 in  unknown file
Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: [10]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[10]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
Rank 23 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23
[10]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Rank 3 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
Rank 10 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10
Rank 1 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] MatSolve_PaStiX line 285 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/pastix/pastix.c
[0]PETSC ERROR: [0] MatSolve line 3104 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] PCApply_LU line 194 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: [0] KSP_PCApply line 228 /mnt/lustre/esd2/esdveng/petsc-3.5.3/include/petsc-private/kspimpl.h
[0]PETSC ERROR: [0] KSPSolve_PREONLY line 19 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/impls/preonly/preonly.c
[0]PETSC ERROR: [0] KSPSolve line 382 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [0] STMatSolve line 143 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsles.c
[0]PETSC ERROR: [0] STApply_Sinvert line 32 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: [0] STApply line 50 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: [0] EPSBasicArnoldi line 47 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylov.c
[0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Default line 179 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/impls/krylov/krylovschur/krylovschur.c
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: ./ex7 on a linux-intel_pastix named nid00828 by esdveng Tue May 26 06:39:09 2015
[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_pastix -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz --download-pastix=/home/proj/14/esdveng/apps/pastix_5.2.2.20.tar.bz2 --download-ptscotch=/home/proj/14/esdveng/apps/scotch_6.0.0_esmumps.tar.gz
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
Rank 0 [Tue May 26 06:39:10 2015] [c4-0c0s15n0] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
_pmiu_daemon(SIGCHLD): [NID 00828] [c4-0c0s15n0] [Tue May 26 06:39:10 2015] PE RANK 15 exit signal Aborted
Application 406154 exit codes: 134
Application 406154 resources: utime ~1s, stime ~2s, Rss ~57484, inblocks ~1676, outblocks ~288

From bsmith at mcs.anl.gov  Tue May 26 07:25:16 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 26 May 2015 07:25:16 -0500
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
In-Reply-To: <87lhgbbyjq.fsf@jedbrown.org>
References: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
	<C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>
	<CAJa9Fi_zpC_7Q-_9H5EMzsqjifMkSw5NaDVHyowJr_eOimz=CQ@mail.gmail.com>
	<87lhgbbyjq.fsf@jedbrown.org>
Message-ID: <9DB2EE53-5F29-4487-962D-F483158AABDC@mcs.anl.gov>


> On May 26, 2015, at 2:14 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> writes:
>> So somehow the optimization flags effect the code execution !?
>> is this a known issue ?
> 
> Either your code is invalid (e.g., undefined behavior), numerically
> unstable, or your vendor has a bug that you should report.

   So first run with valgrind to find any memory issues http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

   If/when that is clean slowly start putting back compiler optimization options (first just something like -O2) until you have a problem.

   Barry

Optimization compiler flags are not all that common but do happen, though most of the time it is a user or PETSc bug


From bsmith at mcs.anl.gov  Tue May 26 07:32:51 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 26 May 2015 07:32:51 -0500
Subject: [petsc-users] Reg :PETSc  Build error with mpi option
In-Reply-To: <AA7F30F772C09F47AE19F45AD9B68B785515CC94@POCITMSEXMB06.LntUniverse.com>
References: <AA7F30F772C09F47AE19F45AD9B68B785515CC94@POCITMSEXMB06.LntUniverse.com>
Message-ID: <DA1DA5E7-ADA3-4867-BF6D-D6AB577B90A4@mcs.anl.gov>


   As the message says: --download-mpich doesn't work under windows. You first need to install MPICH then use that installed version with PETSc. Note that --with-mpich is not a valid option; the option is --with-mpi-dir=/pathtowhereyouinstalledMPI


> On May 26, 2015, at 6:42 AM, Manish Kumar K <Manishkumar.K at LntTechservices.com> wrote:
> 
> Hi PETSc Team,
>  
>                   Greetings of the day!
>  
> I am facing error while building building PETSc using mpich option
>  
> Please let me know where am I going wrong .
>  
>  
> 20063086 at BLRMIESPC-745 /package/petsc-3.5.3
>  
> $ ./configure --with-cc=gcc --with-cxx=g++ --download-mpich="D:/Software/cygwin/package/extpack/mpich-3.1.4" --download-fblaslapack="D:/Software/cygwin/package/                                            extpack/fblaslapack.tar.gz" --with-batch
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
>       Running configure on MPICH; this may take several minutes                                                    
>    ===============================================================================                                                                                                                                                                         *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Error running configure on MPICH.
>    On Microsoft Windows systems, please obtain and install the binary distribution from
>      http://www.mcs.anl.gov/mpi/mpich/mpich-nt
>    then rerun configure.
>    If you choose to install MPICH to a location other than the default, use
>      --with-mpi-dir=<directory>
>    to specify the location of the installation when you rerun configure.
> *******************************************************************************
>  
>  
> The following command also gives me same error
>  
> $ ./configure --with-cc=gcc --with-cxx=g++ --with-mpich=/cygdrive/D/Software/cygwin/package/extpack/mpich-3.1.4.tar.gz  --with-fblaslapack=/cygdrive/D/Software/cygwin/package/                                            extpack/fblaslapack.tar.gz --with-batch
>  
> Help me out rectifying my build issue.
>  
>  
> Thanks & Regards
> Manish K
>  
>  
> L&T Technology Services Ltd
> 
> www.LntTechservices.com
> 
> This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.
> 


From venkateshgk.j at gmail.com  Tue May 26 07:58:24 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Tue, 26 May 2015 18:28:24 +0530
Subject: [petsc-users] Fwd: MUMPS memory error
In-Reply-To: <CABzT7hMMrFiVJ=tkuxshYkVcTUb7N7f5OBkkCW+8c8sCRrwGvA@mail.gmail.com>
References: <CABzT7hMMrFiVJ=tkuxshYkVcTUb7N7f5OBkkCW+8c8sCRrwGvA@mail.gmail.com>
Message-ID: <CABzT7hNiCTK7L=dQ7qr=V_95NOxhhX2Mbks3mRO7JuP76DqQzw@mail.gmail.com>

Any one knows this error, kindly let me know.
Venkatesh
---------- Forwarded message ----------
From: venkatesh g <venkateshgk.j at gmail.com>
Date: Tue, May 26, 2015 at 2:29 PM
Subject: MUMPS memory error
To: mumps-users at listes.ens-lyon.fr


Hi ,

I have a problem with my eigenproblem while using MUMPS along with Petsc
and Slepc. For small matrices I am getting the answer.

When I give large matrices, I am getting the error which is attached as a
log summary.

Kindly help me out.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/09ecef20/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: a7ts.o38614
Type: application/octet-stream
Size: 802728 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/09ecef20/attachment-0001.obj>

From venkateshgk.j at gmail.com  Tue May 26 09:03:49 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Tue, 26 May 2015 19:33:49 +0530
Subject: [petsc-users] MUMPS error and superLU error
Message-ID: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>

I posted a while ago in MUMPS forums but no one seems to reply.

I am solving a large generalized Eigenvalue problem.

I am getting the following error which is attached, after giving the
command:

/cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64 -hosts
compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2 b72t
-st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
-st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
200000

IT IS impossible to allocate so much memory per processor.. it is asking
like around 70 GB per processor.

A serial job in MATLAB for the same matrices takes < 60GB.

After trying out superLU_dist, I have attached the error there also
(segmentation error).

Kindly help me.

Venkatesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/9f294866/attachment-0001.html>
-------------- next part --------------
Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 451597 megabytes

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 457041 megabytes

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[2]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[2]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[8]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64063 megabytes

[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[6]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64043 megabytes

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[5]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[5]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65710 megabytes

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[16]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[16]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[16]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[16]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[16]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64488 megabytes

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[10]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[10]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[10]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[10]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[10]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[10]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[10]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[10]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: Error in external library
[17]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 71786 megabytes

[17]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[17]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[17]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[17]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[17]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[17]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 73270 megabytes

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[32]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 59714 megabytes

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[3]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[3]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[3]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[3]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[3]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[3]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[3]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[3]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[3]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[3]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3
[cli_3]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3
[18]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: Error in external library
[18]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 74466 megabytes

[18]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[18]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[18]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[18]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[18]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[18]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[18]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[18]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[18]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[18]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[18]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[18]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[18]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[18]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[18]PETSC ERROR: [33]PETSC ERROR: [48]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[48]PETSC ERROR: Error in external library
[48]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 79966 megabytes

[48]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[48]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[48]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[48]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[48]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[48]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[48]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[48]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[48]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[48]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[48]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 56809 megabytes

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[7]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[7]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[7]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[7]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[7]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[7]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[7]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[7]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[7]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[7]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 7
[cli_7]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 7
[2]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[2]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[2]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[2]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[2]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[2]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[2]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63961 megabytes

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[1]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[1]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[1]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[1]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[0]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[0]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: Error in external library
[19]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 66131 megabytes

[19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[19]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[19]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[19]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[19]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[19]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[19]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[19]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[19]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[19]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[19]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[19]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[19]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 19
[cli_19]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 19
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[6]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[6]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[6]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[6]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[6]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[6]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[5]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[5]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[5]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[5]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[5]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[5]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[16]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[16]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[16]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[16]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[16]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[16]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[16]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[23]PETSC ERROR: Error in external library
[23]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 67701 megabytes

[23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[23]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[23]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[23]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[23]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[23]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[23]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[23]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[23]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[23]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[23]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[23]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[23]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[23]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 23
[10]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[10]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[17]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[17]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[17]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[17]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[17]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[17]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65476 megabytes

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[20]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[20]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[20]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[20]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[20]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [35]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[35]PETSC ERROR: Error in external library
[35]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57572 megabytes

[35]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[35]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[35]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[35]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[35]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[35]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[35]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[35]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[35]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[35]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[35]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[35]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[35]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[35]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[35]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 35
[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 76742 megabytes

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[14]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[14]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[14]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[14]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[14]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[14]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[14]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[14]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[21]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: Error in external library
[21]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61748 megabytes

[21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[21]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[21]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[21]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[21]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[21]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[21]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[21]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[21]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[21]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[21]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[21]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[21]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[21]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[32]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[32]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[32]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[32]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[32]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[39]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[39]PETSC ERROR: Error in external library
[39]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64773 megabytes

[39]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[39]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[39]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[39]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[39]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[39]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[39]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[39]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[39]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[39]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[39]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[39]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[39]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[39]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[39]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 39
[cli_39]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 39
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 18
[37]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[37]PETSC ERROR: Error in external library
[37]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 55361 megabytes

[37]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[37]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[37]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[37]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[37]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[37]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[40]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[40]PETSC ERROR: Error in external library
[40]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65450 megabytes

[40]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[40]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[40]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[40]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[40]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[40]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[40]PETSC ERROR: [43]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[43]PETSC ERROR: Error in external library
[43]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58608 megabytes

[43]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[43]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[43]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[43]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[43]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[43]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[43]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[43]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[43]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[43]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[43]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[43]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[43]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[43]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[43]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 43
[cli_43]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 43
[22]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: Error in external library
[22]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65695 megabytes

[22]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[22]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[22]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[22]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[22]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[22]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[22]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[22]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[41]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[41]PETSC ERROR: Error in external library
[41]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54285 megabytes

[41]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[41]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[41]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[41]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[41]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[41]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[41]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[41]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[41]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[41]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[41]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[41]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[41]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[41]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[41]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[48]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[48]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[48]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[48]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[55]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[55]PETSC ERROR: Error in external library
[55]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 48241 megabytes

[55]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[55]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[55]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[55]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[55]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[55]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[55]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[55]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[55]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[55]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[55]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[55]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[55]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[55]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[55]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 55
[cli_55]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 55
[34]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[34]PETSC ERROR: Error in external library
[34]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58458 megabytes

[34]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[34]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[34]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[34]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[34]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[34]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[34]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[34]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[34]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[34]PETSC ERROR: [1]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[1]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[1]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[1]PETSC ERROR: [56]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[56]PETSC ERROR: Error in external library
[56]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58551 megabytes

[56]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[56]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[56]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[56]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[56]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[56]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[56]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[56]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[56]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[56]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[56]PETSC ERROR: [42]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[42]PETSC ERROR: Error in external library
[42]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 49860 megabytes

[42]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[42]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[42]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[42]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[42]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[42]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[42]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[42]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[42]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[42]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[42]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[5]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[0]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[0]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[0]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[46]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[46]PETSC ERROR: Error in external library
[46]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 71031 megabytes

[46]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[46]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[46]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[46]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[46]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[46]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[46]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[46]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[46]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[46]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[46]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[46]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[46]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[46]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[46]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[17]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[17]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 17
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 16
[cli_16]: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
[cli_2]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[21]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 21
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[20]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[20]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[20]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[20]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[20]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 20
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
[cli_6]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
[25]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[25]PETSC ERROR: Error in external library
[25]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 68676 megabytes

[25]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[25]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[25]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[25]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[25]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[25]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[25]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[25]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 32
[cli_32]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 32
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 10
[cli_10]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 10
[37]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[37]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[37]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[37]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[37]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[37]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[37]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[37]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[40]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[40]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[40]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[40]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[40]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[40]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[14]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[14]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 14
[cli_14]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 14
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 48
[cli_48]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 48
[cli_18]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 18
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 5
[cli_5]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 5
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[56]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[56]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[56]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[56]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[22]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[22]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[22]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[22]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[22]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[22]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 22
[cli_21]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 21
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[34]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[34]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[34]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[34]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[34]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 34
#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[25]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[25]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[25]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[25]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[25]PETSC ERROR: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 16
[42]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[42]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[42]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[42]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 42
[37]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 37
[cli_20]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 20
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 46
[cli_46]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 46
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 41
[cli_41]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 41
[40]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[40]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 40
[cli_22]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 22
[cli_1]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 56
[cli_56]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 56
[cli_34]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 34
[cli_17]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 17
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
[cli_42]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 42
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[25]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[25]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_40]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 40
[26]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: Error in external library
[26]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58191 megabytes

[26]PETSC ERROR: [cli_37]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 37
[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[26]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[26]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[26]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[26]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 25
[cli_25]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 25
--------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
--------------------- Error Message --------------------------------------------------------------
[52]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[52]PETSC ERROR: Error in external library
[52]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58405 megabytes

[52]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[52]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[52]PETSC ERROR: [26]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[26]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[26]PETSC ERROR: [33]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[52]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[52]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[26]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[26]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Error in external library
[52]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[52]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[52]PETSC ERROR: [26]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[26]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[26]PETSC ERROR: [33]PETSC ERROR: [24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[26]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 46237 megabytes

#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[52]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[52]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[52]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 26
[33]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 69315 megabytes

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[24]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[cli_26]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 26
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[28]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[28]PETSC ERROR: Error in external library
[28]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: [52]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[52]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[52]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[52]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
--------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: Error in external library
[9]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70478 megabytes

[9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[9]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[9]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[9]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[9]PETSC ERROR: [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[24]PETSC ERROR: [30]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 68179 megabytes

[28]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[28]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[28]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[28]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[28]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[38]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[38]PETSC ERROR: Error in external library
[38]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70887 megabytes

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[13]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[13]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63402 megabytes

[38]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[38]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[38]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[38]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[29]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[29]PETSC ERROR: Error in external library
[29]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 66349 megabytes

[29]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[29]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[29]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[29]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[29]PETSC ERROR: [28]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[28]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[28]PETSC ERROR: [38]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[36]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[36]PETSC ERROR: Error in external library
[36]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 67737 megabytes

[36]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[36]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[36]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[36]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[36]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[38]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[38]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[29]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[29]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 52
[cli_52]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 52
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[38]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[38]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[13]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[13]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[24]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[24]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[24]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[38]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[38]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[38]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[29]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[29]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[29]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[28]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[28]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[28]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[28]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[28]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[50]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[50]PETSC ERROR: Error in external library
[50]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61866 megabytes

[50]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[50]PETSC ERROR: [9]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[9]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[9]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[9]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[36]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[36]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[36]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[36]PETSC ERROR: [38]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[38]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[38]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[13]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[13]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[13]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[13]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 24
Petsc Release Version 3.5.3, Jan, 31, 2015 
[50]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[50]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[50]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[29]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[29]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[29]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[28]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[28]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[54]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[54]PETSC ERROR: Error in external library
[54]PETSC ERROR: [9]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[9]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[9]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[36]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[36]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[36]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[36]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 38
[13]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[13]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_24]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 24
[50]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[29]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[29]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 28
[cli_28]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 28
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 49070 megabytes

[54]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[54]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[54]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[54]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[54]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[9]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[9]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[9]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[36]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[36]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_38]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 38
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 13
[cli_13]: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 36
[50]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[50]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[50]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[50]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 29
[cli_36]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 36
[50]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[50]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[50]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 9
[12]PETSC ERROR: [54]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[54]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[54]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[54]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 13
Error in external library
[50]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[50]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_29]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 29
[12]PETSC ERROR: [54]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[54]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[54]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[cli_9]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 9
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63360 megabytes

application called MPI_Abort(MPI_COMM_WORLD, 76) - process 50
Petsc Release Version 3.5.3, Jan, 31, 2015 
[12]PETSC ERROR: [54]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[54]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[54]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[cli_50]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 50
./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[12]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 54
[cli_54]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 54
[33]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Error in external library
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57090 megabytes

#1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
#1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
#8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[30]PETSC ERROR: [33]PETSC ERROR: [12]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[8]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
Error in external library
[30]PETSC ERROR: [33]PETSC ERROR: [8]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[12]PETSC ERROR: [30]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 33
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65248 megabytes

#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[cli_33]: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[30]PETSC ERROR: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 33
[8]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[45]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[45]PETSC ERROR: Error in external library
[45]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 46032 megabytes

[45]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[45]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[45]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[45]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[45]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[45]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[12]PETSC ERROR: [30]PETSC ERROR: [45]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[45]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[45]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[45]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[45]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[45]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[45]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[45]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[45]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[8]PETSC ERROR: [30]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 45
[cli_45]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 45
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[49]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[49]PETSC ERROR: Error in external library
[49]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54280 megabytes

[49]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[49]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[49]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[49]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[49]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[49]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[49]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[30]PETSC ERROR: [57]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[57]PETSC ERROR: Error in external library
[57]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54334 megabytes

[57]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[57]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[57]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[57]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[57]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[57]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[57]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[57]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[57]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[57]PETSC ERROR: [12]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[49]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[49]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[49]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[49]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[49]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[49]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[49]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[49]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 49
[cli_49]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 49
[8]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 30
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[57]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[57]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[57]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[57]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[57]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 57
./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[cli_30]: [cli_57]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 57
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 30
[8]PETSC ERROR: [58]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[58]PETSC ERROR: Error in external library
[58]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 89756 megabytes

[58]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[58]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[58]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[58]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[58]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[58]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[58]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[58]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[58]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[58]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[58]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[58]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[12]PETSC ERROR: [62]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[62]PETSC ERROR: Error in external library
[62]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57502 megabytes

[62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[62]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[62]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[62]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[62]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[62]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[62]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[62]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[62]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[62]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[62]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[62]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[62]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[62]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[62]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 62
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[58]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[58]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[58]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 58
[cli_58]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 58
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61538 megabytes

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[11]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[11]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[11]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[11]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[11]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[11]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[11]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[11]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[11]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 11
[cli_11]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 11
[cli_62]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 62
[8]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61278 megabytes

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[15]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[15]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[15]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[15]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[15]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 15
Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 451597 megabytes

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[0]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Error in external library
[2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 457041 megabytes

[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[2]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[2]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[2]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[2]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Error in external library
[8]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Error in external library
[6]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64063 megabytes

[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[6]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[6]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[6]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[6]PETSC ERROR: [5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Error in external library
[5]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64043 megabytes

[5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[5]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[5]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[5]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[5]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Error in external library
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65710 megabytes

[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[16]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[16]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[16]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[16]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[16]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Error in external library
[10]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64488 megabytes

[10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[10]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[10]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[10]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[10]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[10]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[10]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[10]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[10]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[10]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[10]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[10]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[10]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: Error in external library
[17]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 71786 megabytes

[17]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[17]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[17]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[17]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[17]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[17]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Error in external library
[32]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 73270 megabytes

[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[32]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[32]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[32]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[32]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[32]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[32]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 59714 megabytes

[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[3]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[3]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[3]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[3]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[3]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[3]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[3]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[3]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[3]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[3]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[3]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3
[cli_3]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3
[18]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: Error in external library
[18]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 74466 megabytes

[18]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[18]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[18]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[18]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[18]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[18]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[18]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[18]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[18]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[18]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[18]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[18]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[18]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[18]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[18]PETSC ERROR: [33]PETSC ERROR: [48]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[48]PETSC ERROR: Error in external library
[48]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 79966 megabytes

[48]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[48]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[48]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[48]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[48]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[48]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[48]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[48]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[48]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[48]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[48]PETSC ERROR: [7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Error in external library
[7]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 56809 megabytes

[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[7]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[7]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[7]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[7]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[7]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[7]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[7]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[7]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[7]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[7]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[7]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[7]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[7]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 7
[cli_7]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 7
[2]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[2]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[2]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[2]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[2]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[2]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[2]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63961 megabytes

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[1]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[1]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[1]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[1]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[1]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[0]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[0]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[0]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: Error in external library
[19]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 66131 megabytes

[19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[19]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[19]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[19]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[19]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[19]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[19]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[19]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[19]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[19]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[19]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[19]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[19]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 19
[cli_19]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 19
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[6]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[6]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[6]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[6]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[6]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[6]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[6]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[5]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[5]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[5]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[5]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[5]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[5]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[16]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[16]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[16]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[16]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[16]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[16]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[16]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[23]PETSC ERROR: Error in external library
[23]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 67701 megabytes

[23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[23]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[23]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[23]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[23]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[23]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[23]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[23]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[23]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[23]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[23]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[23]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[23]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[23]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[23]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 23
[10]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[10]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[17]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[17]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[17]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[17]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[17]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[17]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[17]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Error in external library
[20]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65476 megabytes

[20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[20]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[20]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[20]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[20]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[20]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[20]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[20]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[20]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[20]PETSC ERROR: [35]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[35]PETSC ERROR: Error in external library
[35]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57572 megabytes

[35]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[35]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[35]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[35]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[35]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[35]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[35]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[35]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[35]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[35]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[35]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[35]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[35]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[35]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[35]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 35
[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Error in external library
[14]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 76742 megabytes

[14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[14]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[14]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[14]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[14]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[14]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[14]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[14]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[14]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[14]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[14]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[14]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[14]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[21]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: Error in external library
[21]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61748 megabytes

[21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[21]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[21]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[21]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[21]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[21]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[21]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[21]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[21]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[21]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[21]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[21]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[21]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[21]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[32]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[32]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[32]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[32]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[32]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[39]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[39]PETSC ERROR: Error in external library
[39]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 64773 megabytes

[39]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[39]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[39]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[39]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[39]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[39]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[39]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[39]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[39]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[39]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[39]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[39]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[39]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[39]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[39]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 39
[cli_39]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 39
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 18
[37]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[37]PETSC ERROR: Error in external library
[37]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 55361 megabytes

[37]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[37]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[37]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[37]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[37]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[37]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[40]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[40]PETSC ERROR: Error in external library
[40]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65450 megabytes

[40]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[40]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[40]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[40]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[40]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[40]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[40]PETSC ERROR: [43]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[43]PETSC ERROR: Error in external library
[43]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58608 megabytes

[43]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[43]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[43]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[43]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[43]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[43]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[43]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[43]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[43]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[43]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[43]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[43]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[43]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[43]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[43]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 43
[cli_43]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 43
[22]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: Error in external library
[22]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65695 megabytes

[22]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[22]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[22]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[22]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[22]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[22]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[22]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[22]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[22]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[41]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[41]PETSC ERROR: Error in external library
[41]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54285 megabytes

[41]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[41]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[41]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[41]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[41]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[41]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[41]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[41]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[41]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[41]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[41]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[41]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[41]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[41]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[41]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[48]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[48]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[48]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[48]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[55]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[55]PETSC ERROR: Error in external library
[55]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 48241 megabytes

[55]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[55]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[55]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[55]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[55]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[55]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[55]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[55]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[55]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[55]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[55]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[55]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[55]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[55]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[55]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 55
[cli_55]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 55
[34]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[34]PETSC ERROR: Error in external library
[34]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58458 megabytes

[34]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[34]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[34]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[34]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[34]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[34]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[34]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[34]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[34]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[34]PETSC ERROR: [1]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[1]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[1]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[1]PETSC ERROR: [56]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[56]PETSC ERROR: Error in external library
[56]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58551 megabytes

[56]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[56]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[56]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[56]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[56]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[56]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[56]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[56]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[56]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[56]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[56]PETSC ERROR: [42]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[42]PETSC ERROR: Error in external library
[42]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 49860 megabytes

[42]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[42]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[42]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[42]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[42]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[42]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[42]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[42]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[42]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[42]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[42]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[5]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[5]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[0]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[0]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[0]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[0]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[46]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[46]PETSC ERROR: Error in external library
[46]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 71031 megabytes

[46]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[46]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[46]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[46]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[46]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[46]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[46]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[46]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[46]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[46]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[46]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[46]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[46]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[46]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[46]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[17]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[17]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 17
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 16
[cli_16]: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
[cli_2]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 2
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[21]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 21
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[20]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[20]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[20]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[20]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[20]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 20
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
[cli_6]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 6
[25]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[25]PETSC ERROR: Error in external library
[25]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 68676 megabytes

[25]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[25]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[25]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[25]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[25]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[25]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[25]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[25]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 32
[cli_32]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 32
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 10
[cli_10]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 10
[37]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[37]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[37]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[37]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[37]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[37]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[37]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[37]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[40]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[40]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[40]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[40]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[40]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[40]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[14]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[14]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 14
[cli_14]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 14
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 48
[cli_48]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 48
[cli_18]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 18
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 5
[cli_5]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 5
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[56]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[56]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[56]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[56]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[22]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[22]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[22]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[22]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[22]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[22]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 22
[cli_21]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 21
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[34]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[34]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[34]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[34]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[34]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 34
#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[25]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[25]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[25]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[25]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[25]PETSC ERROR: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 16
[42]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[42]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[42]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[42]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 42
[37]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 37
[cli_20]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 20
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 46
[cli_46]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 46
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 41
[cli_41]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 41
[40]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[40]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 40
[cli_22]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 22
[cli_1]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 56
[cli_56]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 56
[cli_34]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 34
[cli_17]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 17
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
[cli_42]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 42
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[25]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[25]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_40]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 40
[26]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: Error in external library
[26]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58191 megabytes

[26]PETSC ERROR: [cli_37]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 37
[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[26]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[26]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[26]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[26]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 25
[cli_25]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 25
--------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
--------------------- Error Message --------------------------------------------------------------
[52]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[52]PETSC ERROR: Error in external library
[52]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 58405 megabytes

[52]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[52]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[52]PETSC ERROR: [26]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[26]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[26]PETSC ERROR: [33]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[52]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[52]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[26]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[26]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Error in external library
[52]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[52]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[52]PETSC ERROR: [26]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[26]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[26]PETSC ERROR: [33]PETSC ERROR: [24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[26]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 46237 megabytes

#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[52]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[52]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[52]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 26
[33]PETSC ERROR: Error in external library
[24]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 69315 megabytes

[24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[24]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[24]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[24]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[cli_26]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 26
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[28]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[28]PETSC ERROR: Error in external library
[28]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: [52]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[52]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[52]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[52]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
--------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: Error in external library
[9]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70478 megabytes

[9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[9]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[9]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[9]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[9]PETSC ERROR: [24]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[24]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[24]PETSC ERROR: [30]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 68179 megabytes

[28]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[28]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[28]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[28]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[28]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[38]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[38]PETSC ERROR: Error in external library
[38]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Error in external library
[13]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 70887 megabytes

[13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[13]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[13]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[13]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[24]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[24]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63402 megabytes

[38]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[38]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[38]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[38]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[29]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[29]PETSC ERROR: Error in external library
[29]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 66349 megabytes

[29]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[29]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[29]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[29]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[29]PETSC ERROR: [28]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[28]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[28]PETSC ERROR: [38]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[13]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[13]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[36]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[36]PETSC ERROR: Error in external library
[36]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 67737 megabytes

[36]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[36]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[36]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[36]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[36]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[38]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[38]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[29]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[29]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 52
[cli_52]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 52
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[38]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[38]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[13]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[13]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[13]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[24]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[24]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[24]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[24]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[38]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[38]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[38]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[29]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[29]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[29]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[28]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[28]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[28]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[28]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[28]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[50]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[50]PETSC ERROR: Error in external library
[50]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61866 megabytes

[50]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[50]PETSC ERROR: [9]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[9]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[9]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[9]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[36]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[36]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[36]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[36]PETSC ERROR: [38]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[38]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[38]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[13]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[13]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[13]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[13]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 24
Petsc Release Version 3.5.3, Jan, 31, 2015 
[50]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[50]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[50]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[29]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[29]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[29]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[28]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[28]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[54]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[54]PETSC ERROR: Error in external library
[54]PETSC ERROR: [9]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[9]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[9]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[36]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[36]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[36]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[36]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 38
[13]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[13]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_24]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 24
[50]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[29]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[29]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 28
[cli_28]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 28
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 49070 megabytes

[54]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[54]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[54]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[54]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[54]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[9]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[9]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[9]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[36]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[36]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_38]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 38
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 13
[cli_13]: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 36
[50]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[50]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[50]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[50]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 29
[cli_36]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 36
[50]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[50]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[50]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 9
[12]PETSC ERROR: [54]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[54]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[54]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[54]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 13
Error in external library
[50]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[50]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[cli_29]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 29
[12]PETSC ERROR: [54]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[54]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[54]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[cli_9]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 9
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 63360 megabytes

application called MPI_Abort(MPI_COMM_WORLD, 76) - process 50
Petsc Release Version 3.5.3, Jan, 31, 2015 
[12]PETSC ERROR: [54]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[54]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[54]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[cli_50]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 50
./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[12]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 54
[cli_54]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 54
[33]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Error in external library
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57090 megabytes

#1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
#4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
#1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[12]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
#8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[30]PETSC ERROR: [33]PETSC ERROR: [12]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[8]PETSC ERROR: [30]PETSC ERROR: [33]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
#5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
Error in external library
[30]PETSC ERROR: [33]PETSC ERROR: [8]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[12]PETSC ERROR: [30]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 33
Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 65248 megabytes

#7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[cli_33]: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[30]PETSC ERROR: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 33
[8]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[45]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[45]PETSC ERROR: Error in external library
[45]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 46032 megabytes

[45]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[45]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[45]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[45]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[45]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[45]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[12]PETSC ERROR: [30]PETSC ERROR: [45]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[45]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[45]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[45]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[45]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[45]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[45]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[45]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[45]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[8]PETSC ERROR: [30]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 45
[cli_45]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 45
#9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[49]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[49]PETSC ERROR: Error in external library
[49]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54280 megabytes

[49]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[49]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[49]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[49]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[49]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[49]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[49]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[30]PETSC ERROR: [57]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[57]PETSC ERROR: Error in external library
[57]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 54334 megabytes

[57]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[57]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[57]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-6.local by venkatesh Wed May 27 00:29:26 2015
[57]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[57]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[57]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[57]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[57]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[57]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[57]PETSC ERROR: [12]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
#3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[49]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[49]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[49]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[49]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[49]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[49]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[49]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[49]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 49
[cli_49]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 49
[8]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 76) - process 30
#6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[57]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[57]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[57]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[57]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[57]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 57
./ex7 on a linux-gnu named compute-0-4.local by venkatesh Wed May 27 00:28:30 2015
[cli_30]: [cli_57]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 57
#10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 30
[8]PETSC ERROR: [58]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[58]PETSC ERROR: Error in external library
[58]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 89756 megabytes

[58]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[58]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[58]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[58]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[58]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[58]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[58]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[58]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[58]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[58]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[58]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[58]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[12]PETSC ERROR: [62]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[62]PETSC ERROR: Error in external library
[62]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 57502 megabytes

[62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[62]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[62]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-7.local by venkatesh Wed May 27 00:28:43 2015
[62]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[62]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[62]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[62]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[62]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[62]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[62]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[62]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[62]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[62]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[62]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[62]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 62
Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[58]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[58]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[58]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 58
[cli_58]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 58
----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Error in external library
[11]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61538 megabytes

[11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[11]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[11]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[11]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[11]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[11]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[11]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[11]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[11]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[11]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[11]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[11]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[11]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[11]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[11]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 11
[cli_11]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 11
[cli_62]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 62
[8]PETSC ERROR: [15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Error in external library
[15]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: Cannot allocate required memory 61278 megabytes

[15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
[15]PETSC ERROR: ./ex7 on a linux-gnu named compute-0-8.local by venkatesh Wed May 27 00:28:08 2015
[15]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=/cluster/share/apps/fblaslapack-3.4.2.tar.gz --download-mpich=/cluster/share/apps/mpich-3.1.tar.gz --download-mumps=/cluster/share/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/cluster/share/apps/scalapack-2.0.2.tgz --download-blacs=/cluster/share/apps/blacs-dev.tar.gz --download-parmetis=/cluster/share/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/cluster/share/apps/metis-5.0.2-p3.tar.gz --download-cmake=/cluster/share/apps/cmake-2.8.12.2.tar.gz --with-scalar-type=complex --with-fortran-kernels=1 --with-large-file-io=1 --with-debugging=no
[15]PETSC ERROR: #1 MatFactorNumeric_MUMPS() line 743 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/impls/aij/mpi/mumps/mumps.c
[15]PETSC ERROR: #2 MatLUFactorNumeric() line 2893 in /cluster/share/venkatesh/petsc-3.5.3/src/mat/interface/matrix.c
[15]PETSC ERROR: #3 PCSetUp_LU() line 152 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: #4 PCSetUp() line 902 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/pc/interface/precon.c
[15]PETSC ERROR: #5 KSPSetUp() line 306 in /cluster/share/venkatesh/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[15]PETSC ERROR: #6 STSetUp_Sinvert() line 145 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/sinvert/sinvert.c
[15]PETSC ERROR: #7 STSetUp() line 301 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15]PETSC ERROR: #8 EPSSetUp() line 207 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[15]PETSC ERROR: #9 EPSSolve() line 88 in /cluster/share/venkatesh/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[15]PETSC ERROR: #10 main() line 146 in /cluster/share/venkatesh/petsc-3.5.3/eigs/ex7.c
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 15
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
[20]PETSC ERROR: [17]PETSC ERROR: ------------------------------------------------------------------------
[20]PETSC ERROR: ------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[17]PETSC ERROR: [20]PETSC ERROR: [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[20]PETSC ERROR: [22]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------
[23]PETSC ERROR: ------------------------------------------------------------------------
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
------------------------------------------------------------------------
[14]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[15]PETSC ERROR: [22]PETSC ERROR: ------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[14]PETSC ERROR: [19]PETSC ERROR: [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[23]PETSC ERROR: ------------------------------------------------------------------------
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[22]PETSC ERROR: [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[21]PETSC ERROR: [23]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[15]PETSC ERROR: [18]PETSC ERROR: ------------------------------------------------------------------------
Try option -start_in_debugger or -on_error_attach_debugger
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[14]PETSC ERROR: [19]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: [21]PETSC ERROR: ------------------------------------------------------------------------
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
Try option -start_in_debugger or -on_error_attach_debugger
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[22]PETSC ERROR: [23]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[18]PETSC ERROR: [19]PETSC ERROR: [21]PETSC ERROR: [20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[22]PETSC ERROR: likely location of problem given in stack below
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[20]PETSC ERROR: [21]PETSC ERROR: [18]PETSC ERROR: [19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
---------------------  Stack Frames ------------------------------------
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
Try option -start_in_debugger or -on_error_attach_debugger
[20]PETSC ERROR: [14]PETSC ERROR: [19]PETSC ERROR: [18]PETSC ERROR: likely location of problem given in stack below
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
Note: The EXACT line numbers in the stack are not available,
[14]PETSC ERROR: [15]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[18]PETSC ERROR: likely location of problem given in stack below
[14]PETSC ERROR: [20]PETSC ERROR: [15]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
      INSTEAD the line number of the start of the function
[22]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: likely location of problem given in stack below
likely location of problem given in stack below
      is given.
---------------------  Stack Frames ------------------------------------
[14]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: [15]PETSC ERROR:       INSTEAD the line number of the start of the function
[22]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[15]PETSC ERROR:       is given.
[20] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
      INSTEAD the line number of the start of the function
[14]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: [19]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
Note: The EXACT line numbers in the stack are not available,
[20] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
likely location of problem given in stack below
[22]PETSC ERROR: [21]PETSC ERROR: [20]PETSC ERROR:       INSTEAD the line number of the start of the function
Note: The EXACT line numbers in the stack are not available,
[20] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[21]PETSC ERROR: [22]PETSC ERROR: [15]PETSC ERROR: [14] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
      is given.
      INSTEAD the line number of the start of the function
[20]PETSC ERROR:       is given.
[22]PETSC ERROR: [20] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[21]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR:       is given.
[21] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[22]PETSC ERROR: [14] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[15] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[14]PETSC ERROR: [22] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[21]PETSC ERROR: [20]PETSC ERROR: [22]PETSC ERROR: [14] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[15]PETSC ERROR: [21] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[20] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[21]PETSC ERROR: [22] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[14]PETSC ERROR: [15] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[14] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[22]PETSC ERROR: [14]PETSC ERROR: [15]PETSC ERROR: [14] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[20]PETSC ERROR: [15] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[14]PETSC ERROR: [20] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[21] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[16]PETSC ERROR: [22] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[21]PETSC ERROR: [20]PETSC ERROR: [21] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
------------------------------------------------------------------------
[14] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15]PETSC ERROR: [16]PETSC ERROR: [22]PETSC ERROR: [21]PETSC ERROR: [20] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[15] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[14]PETSC ERROR: [21] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[20]PETSC ERROR: [22] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[21]PETSC ERROR: [20] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[21] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[14] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[15]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[21]PETSC ERROR: [22]PETSC ERROR: [14]PETSC ERROR: [15] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[21] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[16]PETSC ERROR: [14] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[22] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[15] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[22]PETSC ERROR: [21]PETSC ERROR: [22] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[15]PETSC ERROR: [16]PETSC ERROR: [21] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[15] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[22]PETSC ERROR: [16]PETSC ERROR: [22] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[22]PETSC ERROR: [22] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[17]PETSC ERROR: [20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: [17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
--------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: [20]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
Signal received
--------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[23]PETSC ERROR: Signal received
--------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: likely location of problem given in stack below
Signal received
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[15]PETSC ERROR: [20]PETSC ERROR: [21]PETSC ERROR: Signal received
[14]PETSC ERROR: [23]PETSC ERROR: [15]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[18]PETSC ERROR: [15]PETSC ERROR: [14]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
likely location of problem given in stack below
[15]PETSC ERROR: ./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[18]PETSC ERROR: ./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[14]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[15]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[14]PETSC ERROR: [17]PETSC ERROR: [15]PETSC ERROR: likely location of problem given in stack below
#1 User provided function() line 0 in  unknown file
#1 User provided function() line 0 in  unknown file
[17]PETSC ERROR: [19]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[22]PETSC ERROR: [17]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
--------------------- Error Message --------------------------------------------------------------
Note: The EXACT line numbers in the stack are not available,
[19]PETSC ERROR: [22]PETSC ERROR: [17]PETSC ERROR: Signal received
Note: The EXACT line numbers in the stack are not available,
      INSTEAD the line number of the start of the function
[22]PETSC ERROR: [19]PETSC ERROR: [17]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
      INSTEAD the line number of the start of the function
      is given.
[22]PETSC ERROR: [19]PETSC ERROR: [17]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
      is given.
[17] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[19]PETSC ERROR: [22]PETSC ERROR: [17]PETSC ERROR: [19] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[17] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[19]PETSC ERROR: [22]PETSC ERROR: [17]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[19] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[17] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[19]PETSC ERROR: [22]PETSC ERROR: [17]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[16]PETSC ERROR: [17] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[19] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
---------------------  Stack Frames ------------------------------------
[19]PETSC ERROR: likely location of problem given in stack below
[17]PETSC ERROR: [19] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: [17] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[19]PETSC ERROR: [17]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
[19] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[17] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[16]PETSC ERROR: [19]PETSC ERROR: [17]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[19] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[17] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[16]PETSC ERROR: [19]PETSC ERROR:       INSTEAD the line number of the start of the function
[17]PETSC ERROR: [16]PETSC ERROR: [19] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[17] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
      is given.
[19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
Rank 14 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14
Rank 15 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15
[16]PETSC ERROR: [21]PETSC ERROR: [16] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
Petsc Release Version 3.5.3, Jan, 31, 2015 
[21]PETSC ERROR: [16]PETSC ERROR: [20]PETSC ERROR: ./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[16] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[21]PETSC ERROR: [16]PETSC ERROR: [17]PETSC ERROR: Rank 22 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[21]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[16]PETSC ERROR: [17]PETSC ERROR: [20]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[16] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
Signal received
#1 User provided function() line 0 in  unknown file
[17]PETSC ERROR: [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[18]PETSC ERROR: [16] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[17]PETSC ERROR: [16]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
Petsc Release Version 3.5.3, Jan, 31, 2015 
[16] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
[18]PETSC ERROR: [17]PETSC ERROR:       INSTEAD the line number of the start of the function
[16]PETSC ERROR: [18]PETSC ERROR: ./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[16] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
      is given.
[16]PETSC ERROR: [17]PETSC ERROR: [16] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[18]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[18] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[23]PETSC ERROR: [17]PETSC ERROR: [18]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
#1 User provided function() line 0 in  unknown file
[18] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[23]PETSC ERROR: [18]PETSC ERROR:       INSTEAD the line number of the start of the function
[19]PETSC ERROR: [18] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
      is given.
[18]PETSC ERROR: [19]PETSC ERROR: [18] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[23]PETSC ERROR: Signal received
[18]PETSC ERROR: [23] MatLUFactorNumeric_SuperLU_DIST line 273 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
Rank 20 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20
Rank 21 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21
[23]PETSC ERROR: [19]PETSC ERROR: [18] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
[23] MatLUFactorNumeric line 2882 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/mat/interface/matrix.c
[16]PETSC ERROR: [23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[18]PETSC ERROR: [23] PCSetUp_LU line 99 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/pc/impls/factor/lu/lu.c
[19]PETSC ERROR: [18] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Petsc Release Version 3.5.3, Jan, 31, 2015 
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: [18]PETSC ERROR: [23] KSPSetUp line 220 /mnt/lustre/esd2/esdveng/petsc-3.5.3/src/ksp/ksp/interface/itfunc.c
[18] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[23]PETSC ERROR: [16]PETSC ERROR: [23] STSetUp_Shift line 103 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/impls/shift/shift.c
./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[18]PETSC ERROR: [23]PETSC ERROR: [19]PETSC ERROR: [18] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
[23] STSetUp line 269 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/sys/classes/st/interface/stsolve.c
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
Signal received
[19]PETSC ERROR: [23]PETSC ERROR: [16]PETSC ERROR: #1 User provided function() line 0 in  unknown file
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[23] EPSSetUp line 58 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssetup.c
[16]PETSC ERROR: [23]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Rank 17 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17
[16]PETSC ERROR: [23] EPSSolve line 83 /mnt/lustre/esd2/esdveng/petsc-3.5.3/slepc-3.5.3/src/eps/interface/epssolve.c
./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[16]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[18]PETSC ERROR: [16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
#1 User provided function() line 0 in  unknown file
[18]PETSC ERROR: Signal received
[18]PETSC ERROR: [23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
--------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: [23]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Signal received
[18]PETSC ERROR: [23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
[23]PETSC ERROR: [18]PETSC ERROR: Petsc Release Version 3.5.3, Jan, 31, 2015 
Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[23]PETSC ERROR: [18]PETSC ERROR: ./ex7 on a linux-intel_superlu named nid01474 by esdveng Tue May 26 08:14:47 2015
#1 User provided function() line 0 in  unknown file
[23]PETSC ERROR: Configure options --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel_superlu -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/mnt/lustre/esd2/esdveng/valgrind --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-superlu_dist=/home/proj/14/esdveng/apps/superlu_dist_3.3.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz
[23]PETSC ERROR: Rank 19 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19
#1 User provided function() line 0 in  unknown file
Rank 16 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16
Rank 18 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18
Rank 23 [Tue May 26 08:14:47 2015] [c7-0c2s0n2] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23
_pmiu_daemon(SIGCHLD): [NID 01474] [c7-0c2s0n2] [Tue May 26 08:14:47 2015] PE RANK 14 exit signal Aborted
[NID 01474] 2015-05-26 18:46:18 Apid 406256: initiated application termination
Application 406256 exit codes: 134
Application 406256 resources: utime ~0s, stime ~2s, Rss ~13516, inblocks ~1675, outblocks ~287

From hzhang at mcs.anl.gov  Tue May 26 10:32:54 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 26 May 2015 10:32:54 -0500
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
Message-ID: <CAGCphBuh9Nco8SvQhfsLC9AYnt9Uj_Nfyy_muaSxJRGhspdsUg@mail.gmail.com>

'A serial job in MATLAB for the same matrices takes < 60GB. '
Can you run this case in serial? If so, try petsc, superlu or mumps to make
sure the matrix is non-singular.

Both mumps and superlu_dist show crash in MatFactorNumeric(). Mumps gives
error
[16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
Cannot allocate required memory 65710 megabytes.

Does your code work for smaller problems?
Try using more processors?

Why use such huge '-mat_mumps_icntl_14 200000' (percentage of estimated
workspace increase)? The default is 20. Try 40?

Superlu_dist usually uses less memory than mumps, but it also crashes. I
guess something wrong with your matrix. Is it singular?
Run superlu_dist on a slightly smaller matrix with option
'-mat_superlu_dist_statprint' which displays memory usage info., e.g.,

petsc/src/ksp/ksp/examples/tutorials (maint)
$ mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package superlu_dist
-mat_superlu_dist_statprint
Nonzeros in L       560
Nonzeros in U       560
nonzeros in L+U     1064
nonzeros in LSUB    248
NUMfact space (MB) sum(procs):  L\U 0.01 all 0.05
Total highmark (MB):  All 0.05 Avg 0.02 Max 0.02
EQUIL time             0.00
ROWPERM time           0.00
COLPERM time           0.00
SYMBFACT time          0.00
DISTRIBUTE time        0.00
FACTOR time            0.00
Factor flops 1.181000e+04 Mflops     4.80
SOLVE time             0.00
SOLVE time             0.00
Solve flops 2.194000e+03 Mflops     4.43
SOLVE time             0.00
Solve flops 2.194000e+03 Mflops     5.14
Norm of error 1.18018e-15 iterations 1

Hong


On Tue, May 26, 2015 at 9:03 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> I posted a while ago in MUMPS forums but no one seems to reply.
>
> I am solving a large generalized Eigenvalue problem.
>
> I am getting the following error which is attached, after giving the
> command:
>
> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64 -hosts
> compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2 b72t
> -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
> 200000
>
> IT IS impossible to allocate so much memory per processor.. it is asking
> like around 70 GB per processor.
>
> A serial job in MATLAB for the same matrices takes < 60GB.
>
> After trying out superLU_dist, I have attached the error there also
> (segmentation error).
>
> Kindly help me.
>
> Venkatesh
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/e9c3a487/attachment.html>

From vijay.gopal.c at gmail.com  Tue May 26 10:54:23 2015
From: vijay.gopal.c at gmail.com (Vijay Gopal Chilkuri)
Date: Tue, 26 May 2015 17:54:23 +0200
Subject: [petsc-users] ERROR: Scalar value must be same on all processes
In-Reply-To: <9DB2EE53-5F29-4487-962D-F483158AABDC@mcs.anl.gov>
References: <CAJa9Fi9EAMM_6D-GhPcE8mM-b5+7v_5Df9WXEj1BB_XJN4Yquw@mail.gmail.com>
	<C11F9100-7C06-4DF8-A843-DFAEFD9018D7@mcs.anl.gov>
	<CAJa9Fi_zpC_7Q-_9H5EMzsqjifMkSw5NaDVHyowJr_eOimz=CQ@mail.gmail.com>
	<87lhgbbyjq.fsf@jedbrown.org>
	<9DB2EE53-5F29-4487-962D-F483158AABDC@mcs.anl.gov>
Message-ID: <CAJa9Fi8mep1S9+A7e4OW3k7XhFbWmOQjK-LTUhsH+N45H0Y=og@mail.gmail.com>

I did what barry adviced.
I found that it's only the -xAVX option which causes problems.

If I compile the code with the above COPTFLAGS , *excluding* the *-xAVX*
everything works fine !

Thanks for the advice.

On Tue, May 26, 2015 at 2:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 26, 2015, at 2:14 AM, Jed Brown <jed at jedbrown.org> wrote:
> >
> > Vijay Gopal Chilkuri <vijay.gopal.c at gmail.com> writes:
> >> So somehow the optimization flags effect the code execution !?
> >> is this a known issue ?
> >
> > Either your code is invalid (e.g., undefined behavior), numerically
> > unstable, or your vendor has a bug that you should report.
>
>    So first run with valgrind to find any memory issues
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
>    If/when that is clean slowly start putting back compiler optimization
> options (first just something like -O2) until you have a problem.
>
>    Barry
>
> Optimization compiler flags are not all that common but do happen, though
> most of the time it is a user or PETSc bug
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/8e7e6bf1/attachment.html>

From balay at mcs.anl.gov  Tue May 26 11:46:28 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 26 May 2015 11:46:28 -0500
Subject: [petsc-users] petsc-3.5.4.tar.gz now available
Message-ID: <alpine.LFD.2.11.1505261145080.4830@asterix>

Dear PETSc users,

The patch release petsc-3.5.4 is now available for download.
http://www.mcs.anl.gov/petsc/download/index.html

Some of the changes include:

 * Mat: Fix issue where the nonzerostate of Mat_MPIAIJ did not
   monotonically increase when matrix entries are zeroed in B.

 * Mat: bugfix for MatPtAPSymbolic_MPIAIJ_MPIAIJ()

 * DMDA: DMDAVTKWriteAll_VTS - Fix handling of higher-dimension
   coordinates

 * MatNestGetISs: use custom fortranstub to handle PETSC_NULL_OBJECT

 * Sys: fix PetscSubcommSetTypeGeneral wrt zero-ing the subsizes array

 * PetscEnumFind: Fix wrong counting of entries in list

 * configure: fix failure with --with-batch=1

 * MatLoad_SeqDense() always allocated new space for the data even if
   the user already provided it

 * Added PetscBagRegister64bitInt()

 * Removed direct use of HDF5_NATIVE_DOUBLE to support single
   precision as well

 * DMShell: destroy local to local scatter

 * Fix for ISGlobalToLocalMappingApply() for when it has a block size > 1

 * bugfix for MatGetRowIJ_SeqSBAIJ()

 * PetscDrawLGDraw: draw and flush only in process zero

 * PetscDrawSetType: Check the draw type is registered before destroying

 * TS: Use %g to draw time in TSMonitorDrawSolution()

 * PCMG must set initial guess nonzero for down (pre) smoother when
   using full or Kaskade multigrid because otherwise it zeros out the
   interpolated solution from the coarser level

 * Ensure the DMDA PetscViewerBinary writer using MPI-IO functionality
   respects the skip header flag when wither writing or reading files.

 * Added checks to enable loading of a vector which was written
   without the binary header.

Satish


From hsahasra at purdue.edu  Tue May 26 12:42:44 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Tue, 26 May 2015 13:42:44 -0400
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
Message-ID: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>

Hi,

I'm using LibMesh for meshing a non-linear equation. I want to use
NRichardson non-linear solver to solve the equation. I have a Newton trust
region non-linear solver as a pre-conditioner. Both these SNES are created
via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.

I need to re-init the LibMesh NonlinearImplicitSystem each time the
preconditioner is applied. Also, I need the preconditioner to be solved via
the LibMesh wrapper instead of directly through SNESSolve. How can make it
work this way? Any help is appreciated!

Thanks,
Harshad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/adc8905c/attachment.html>

From knepley at gmail.com  Tue May 26 14:25:24 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 26 May 2015 14:25:24 -0500
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
Message-ID: <CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>

On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu>
wrote:

> Hi,
>
> I'm using LibMesh for meshing a non-linear equation. I want to use
> NRichardson non-linear solver to solve the equation. I have a Newton trust
> region non-linear solver as a pre-conditioner. Both these SNES are created
> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>
> I need to re-init the LibMesh NonlinearImplicitSystem each time the
> preconditioner is applied. Also, I need the preconditioner to be solved via
> the LibMesh wrapper instead of directly through SNESSolve. How can make it
> work this way? Any help is appreciated!
>

I think the easiest way is to have them both look like SNES objects. Then
you can just call SNESSetNPC() or use -snes_npc_side left -npc_snes_type
newtonls

Can you explain what the LibMesh wrapper is doing that you you need? We
could wrap that object in a SNES, but it sounds
simpler to me to unwrap it.

  Thanks,

     Matt


> Thanks,
> Harshad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/5d0146f5/attachment.html>

From hsahasra at purdue.edu  Tue May 26 15:07:36 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Tue, 26 May 2015 16:07:36 -0400
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
	<CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
Message-ID: <CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>

Hi Matt,

Actually I just checked, I don't need to re-init the preconditioner LibMesh
solver.

The wrapper is interfaced to the mesh and makes it easier to assemble the
system matrices. It also keeps track of constraints. The wrapper
function PetscNonlinearSolver::solve
updates the system vectors and matrices, then calls SNESSolve. Thus, I need
to call PetscNonlinearSolver::solve instead of SNESSolve in SNESApplyNPC.

Do you mean wrap it in SNESSHELL?

Thanks,
Harshad

On Tue, May 26, 2015 at 3:25 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <
> hsahasra at purdue.edu> wrote:
>
>> Hi,
>>
>> I'm using LibMesh for meshing a non-linear equation. I want to use
>> NRichardson non-linear solver to solve the equation. I have a Newton trust
>> region non-linear solver as a pre-conditioner. Both these SNES are created
>> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>>
>> I need to re-init the LibMesh NonlinearImplicitSystem each time the
>> preconditioner is applied. Also, I need the preconditioner to be solved via
>> the LibMesh wrapper instead of directly through SNESSolve. How can make it
>> work this way? Any help is appreciated!
>>
>
> I think the easiest way is to have them both look like SNES objects. Then
> you can just call SNESSetNPC() or use -snes_npc_side left -npc_snes_type
> newtonls
>
> Can you explain what the LibMesh wrapper is doing that you you need? We
> could wrap that object in a SNES, but it sounds
> simpler to me to unwrap it.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Harshad
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/0f3c46c7/attachment.html>

From mfadams at lbl.gov  Tue May 26 15:24:22 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 27 May 2015 04:24:22 +0800
Subject: [petsc-users] TS algo question
Message-ID: <CADOhEh4kFBGH37WCX_0OcrKPeg-R=VWH4+JxXLEH9R1Zg860pw@mail.gmail.com>

I have setup the equations attached using TSSetIJacobian and
 TSSetIFunction.

This is intended to be a fully implicit solvers with some auxiliary
variables.

We want to verify the code with the current forward Euler solver.  Are
there command line that would give me a solver that solves the two
auxiliary variables (I do have them ordered first).  Or do I need to
reconfigure the solver somehow?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/e26b787a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: XGC-EM-4field-Fast.pdf
Type: application/pdf
Size: 82887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/e26b787a/attachment-0001.pdf>

From knepley at gmail.com  Tue May 26 15:25:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 26 May 2015 15:25:56 -0500
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
	<CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
	<CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>
Message-ID: <CAMYG4G=+gNFjhMr6xWjBRSS8NXeNa5jpE_b_uXRvwHnkAR_nHA@mail.gmail.com>

On Tue, May 26, 2015 at 3:07 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu>
wrote:

> Hi Matt,
>
> Actually I just checked, I don't need to re-init the preconditioner
> LibMesh solver.
>
> The wrapper is interfaced to the mesh and makes it easier to assemble the
> system matrices. It also keeps track of constraints. The wrapper function PetscNonlinearSolver::solve
> updates the system vectors and matrices, then calls SNESSolve. Thus, I
> need to call PetscNonlinearSolver::solve instead of SNESSolve in
> SNESApplyNPC.
>
> Do you mean wrap it in SNESSHELL?
>

Yes, exactly.

  Thanks,

     Matt


> Thanks,
> Harshad
>
> On Tue, May 26, 2015 at 3:25 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <
>> hsahasra at purdue.edu> wrote:
>>
>>> Hi,
>>>
>>> I'm using LibMesh for meshing a non-linear equation. I want to use
>>> NRichardson non-linear solver to solve the equation. I have a Newton trust
>>> region non-linear solver as a pre-conditioner. Both these SNES are created
>>> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>>>
>>> I need to re-init the LibMesh NonlinearImplicitSystem each time the
>>> preconditioner is applied. Also, I need the preconditioner to be solved via
>>> the LibMesh wrapper instead of directly through SNESSolve. How can make it
>>> work this way? Any help is appreciated!
>>>
>>
>> I think the easiest way is to have them both look like SNES objects. Then
>> you can just call SNESSetNPC() or use -snes_npc_side left -npc_snes_type
>> newtonls
>>
>> Can you explain what the LibMesh wrapper is doing that you you need? We
>> could wrap that object in a SNES, but it sounds
>> simpler to me to unwrap it.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Harshad
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/34212982/attachment.html>

From knepley at gmail.com  Tue May 26 15:37:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 26 May 2015 15:37:47 -0500
Subject: [petsc-users] Pastix with Petsc and Slepc
In-Reply-To: <CABzT7hOCQtiQxXfcTK7euA=cTaTPcZrcGzQmnU_Ey6vk_23iww@mail.gmail.com>
References: <CABzT7hOCQtiQxXfcTK7euA=cTaTPcZrcGzQmnU_Ey6vk_23iww@mail.gmail.com>
Message-ID: <CAMYG4GkZcG1GaGQcNs2HtXP3i4nQ21SzKYy6TUBm9_X+DnDGxA@mail.gmail.com>

Have you tried on a single process?

  Thanks,

    Matt

On Tue, May 26, 2015 at 7:01 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi I am trying to solve my eigenproblem Ax=lambdaBx with a direct linear
> solver Pastix
>
> However I get this error (attached). It works without the pastix option.
>
> My matrices are in binary format.
>
> I gave the command : aprun -n 24 -N 24 ./ex7 -f1 a20t -f2 b20t -st_type
> sinvert -eps_nev 3 -log_summary -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package pastix -ksp_monitor
>
> Kindly help me out.
>
> Venkatesh
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150526/d4af1e87/attachment.html>

From hengjiew at uci.edu  Tue May 26 16:18:46 2015
From: hengjiew at uci.edu (frank)
Date: Tue, 26 May 2015 14:18:46 -0700
Subject: [petsc-users] Question about using Hypre with OpenMP under Petsc
Message-ID: <5564E336.4050106@uci.edu>

Hi

I am trying to use multigrid to solve a large sparse linear system. I 
use Hypre boomeramg as the preconditioner. The code calling KSPSolve is 
paralleled by MPI.

I want to set Hypre to use OpenMP. Here is what I did:
* I downloaded and compiled Hypre through Petsc
* I recompiled the Hypre with " --with-opemp ".
* I set "-pc_type hypre" and "-pc_type_type boomeramg" for Petsc.

My question:
? In this way, would Hypre use OpenMP to parallel the execution when 
KSPSolve is called ?
? If this does not work, is there another way I can set Hypre to use 
OpenMP under Petsc ?
? Is there a way I can know explicitly whether Hypre is using OpenMP 
under Petsc or not ?

Thank you so much
Frank





From bsmith at mcs.anl.gov  Tue May 26 16:41:32 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 26 May 2015 16:41:32 -0500
Subject: [petsc-users] Question about using Hypre with OpenMP under Petsc
In-Reply-To: <5564E336.4050106@uci.edu>
References: <5564E336.4050106@uci.edu>
Message-ID: <336DD41E-118A-4400-9E7E-7974859550DF@mcs.anl.gov>


> On May 26, 2015, at 4:18 PM, frank <hengjiew at uci.edu> wrote:
> 
> Hi
> 
> I am trying to use multigrid to solve a large sparse linear system. I use Hypre boomeramg as the preconditioner. The code calling KSPSolve is paralleled by MPI.
> 
> I want to set Hypre to use OpenMP. Here is what I did:
> * I downloaded and compiled Hypre through Petsc
> * I recompiled the Hypre with " --with-opemp ".

   Ok, you need to make sure that PETSc is linking against the OpenMP compiled version of hypre libraries. Use ldd on linux or otool -L on Mac.

> * I set "-pc_type hypre" and "-pc_type_type boomeramg" for Petsc.
> 
> My question:
> ? In this way, would Hypre use OpenMP to parallel the execution when KSPSolve is called ?
> ? If this does not work, is there another way I can set Hypre to use OpenMP under Petsc ?
> ? Is there a way I can know explicitly whether Hypre is using OpenMP under Petsc or not ?

   Your question really has little to do with PETSc and more to do with hypre. You need to look through the hypre documentation and find out how you control the number of OpenMP threads that hypre uses (likely it is some environmental variables).  Then run varying this number of threads and see what happens, if you use more threads does it go faster?  It is best to make this test with just a single MPI process and 1,2,4, 8 OpenMP threads.

  Barry




   


> 
> Thank you so much
> Frank
> 
> 
> 
> 


From hengjiew at uci.edu  Tue May 26 18:52:38 2015
From: hengjiew at uci.edu (frank)
Date: Tue, 26 May 2015 16:52:38 -0700
Subject: [petsc-users] Question about using Hypre with OpenMP under Petsc
In-Reply-To: <336DD41E-118A-4400-9E7E-7974859550DF@mcs.anl.gov>
References: <5564E336.4050106@uci.edu>
	<336DD41E-118A-4400-9E7E-7974859550DF@mcs.anl.gov>
Message-ID: <55650746.2090809@uci.edu>

Hi Barry,

Thank you for your prompt reply.
Which executable lib should I use ldd to check?

Thank you,
Frank.

On 05/26/2015 02:41 PM, Barry Smith wrote:
>> On May 26, 2015, at 4:18 PM, frank <hengjiew at uci.edu> wrote:
>>
>> Hi
>>
>> I am trying to use multigrid to solve a large sparse linear system. I use Hypre boomeramg as the preconditioner. The code calling KSPSolve is paralleled by MPI.
>>
>> I want to set Hypre to use OpenMP. Here is what I did:
>> * I downloaded and compiled Hypre through Petsc
>> * I recompiled the Hypre with " --with-opemp ".
>     Ok, you need to make sure that PETSc is linking against the OpenMP compiled version of hypre libraries. Use ldd on linux or otool -L on Mac.
>
>> * I set "-pc_type hypre" and "-pc_type_type boomeramg" for Petsc.
>>
>> My question:
>> ? In this way, would Hypre use OpenMP to parallel the execution when KSPSolve is called ?
>> ? If this does not work, is there another way I can set Hypre to use OpenMP under Petsc ?
>> ? Is there a way I can know explicitly whether Hypre is using OpenMP under Petsc or not ?
>     Your question really has little to do with PETSc and more to do with hypre. You need to look through the hypre documentation and find out how you control the number of OpenMP threads that hypre uses (likely it is some environmental variables).  Then run varying this number of threads and see what happens, if you use more threads does it go faster?  It is best to make this test with just a single MPI process and 1,2,4, 8 OpenMP threads.
>
>    Barry
>
>
>
>
>     
>
>
>> Thank you so much
>> Frank
>>
>>
>>
>>


From bsmith at mcs.anl.gov  Tue May 26 19:53:25 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 26 May 2015 19:53:25 -0500
Subject: [petsc-users] Question about using Hypre with OpenMP under Petsc
In-Reply-To: <55650746.2090809@uci.edu>
References: <5564E336.4050106@uci.edu>
	<336DD41E-118A-4400-9E7E-7974859550DF@mcs.anl.gov>
	<55650746.2090809@uci.edu>
Message-ID: <8D5AF354-BA19-4135-A4FE-8CC48CB4DF21@mcs.anl.gov>


  $PETSC_DIR/$PETSC_ARCH/lib/libpetsc*


> On May 26, 2015, at 6:52 PM, frank <hengjiew at uci.edu> wrote:
> 
> Hi Barry,
> 
> Thank you for your prompt reply.
> Which executable lib should I use ldd to check?
> 
> Thank you,
> Frank.
> 
> On 05/26/2015 02:41 PM, Barry Smith wrote:
>>> On May 26, 2015, at 4:18 PM, frank <hengjiew at uci.edu> wrote:
>>> 
>>> Hi
>>> 
>>> I am trying to use multigrid to solve a large sparse linear system. I use Hypre boomeramg as the preconditioner. The code calling KSPSolve is paralleled by MPI.
>>> 
>>> I want to set Hypre to use OpenMP. Here is what I did:
>>> * I downloaded and compiled Hypre through Petsc
>>> * I recompiled the Hypre with " --with-opemp ".
>>    Ok, you need to make sure that PETSc is linking against the OpenMP compiled version of hypre libraries. Use ldd on linux or otool -L on Mac.
>> 
>>> * I set "-pc_type hypre" and "-pc_type_type boomeramg" for Petsc.
>>> 
>>> My question:
>>> ? In this way, would Hypre use OpenMP to parallel the execution when KSPSolve is called ?
>>> ? If this does not work, is there another way I can set Hypre to use OpenMP under Petsc ?
>>> ? Is there a way I can know explicitly whether Hypre is using OpenMP under Petsc or not ?
>>    Your question really has little to do with PETSc and more to do with hypre. You need to look through the hypre documentation and find out how you control the number of OpenMP threads that hypre uses (likely it is some environmental variables).  Then run varying this number of threads and see what happens, if you use more threads does it go faster?  It is best to make this test with just a single MPI process and 1,2,4, 8 OpenMP threads.
>> 
>>   Barry
>> 
>> 
>> 
>> 
>>    
>> 
>>> Thank you so much
>>> Frank
>>> 
>>> 
>>> 
>>> 
> 


From venkateshgk.j at gmail.com  Wed May 27 06:08:04 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Wed, 27 May 2015 16:38:04 +0530
Subject: [petsc-users] Pastix with Petsc and Slepc
In-Reply-To: <CAMYG4GkZcG1GaGQcNs2HtXP3i4nQ21SzKYy6TUBm9_X+DnDGxA@mail.gmail.com>
References: <CABzT7hOCQtiQxXfcTK7euA=cTaTPcZrcGzQmnU_Ey6vk_23iww@mail.gmail.com>
	<CAMYG4GkZcG1GaGQcNs2HtXP3i4nQ21SzKYy6TUBm9_X+DnDGxA@mail.gmail.com>
Message-ID: <CABzT7hPKcCPsJVeq+XwxCcozLB=U4gWfTxnfqLhhw7_F1LJd=g@mail.gmail.com>

Yes. I did try on a single core. It gives same error for any size of the
matrix.

PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably
memory access out of range

Kindly let me know.

Venkatesh

On Wed, May 27, 2015 at 2:07 AM, Matthew Knepley <knepley at gmail.com> wrote:

> Have you tried on a single process?
>
>   Thanks,
>
>     Matt
>
> On Tue, May 26, 2015 at 7:01 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi I am trying to solve my eigenproblem Ax=lambdaBx with a direct linear
>> solver Pastix
>>
>> However I get this error (attached). It works without the pastix option.
>>
>> My matrices are in binary format.
>>
>> I gave the command : aprun -n 24 -N 24 ./ex7 -f1 a20t -f2 b20t -st_type
>> sinvert -eps_nev 3 -log_summary -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package pastix -ksp_monitor
>>
>> Kindly help me out.
>>
>> Venkatesh
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/d01e4e1d/attachment-0001.html>

From knepley at gmail.com  Wed May 27 08:32:37 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 08:32:37 -0500
Subject: [petsc-users] Pastix with Petsc and Slepc
In-Reply-To: <CABzT7hPKcCPsJVeq+XwxCcozLB=U4gWfTxnfqLhhw7_F1LJd=g@mail.gmail.com>
References: <CABzT7hOCQtiQxXfcTK7euA=cTaTPcZrcGzQmnU_Ey6vk_23iww@mail.gmail.com>
	<CAMYG4GkZcG1GaGQcNs2HtXP3i4nQ21SzKYy6TUBm9_X+DnDGxA@mail.gmail.com>
	<CABzT7hPKcCPsJVeq+XwxCcozLB=U4gWfTxnfqLhhw7_F1LJd=g@mail.gmail.com>
Message-ID: <CAMYG4GkKfs=keaNJhd7K2wr2jO8oS0XFautr3aqdWYoh8xjxmA@mail.gmail.com>

On Wed, May 27, 2015 at 6:08 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Yes. I did try on a single core. It gives same error for any size of the
> matrix.
>
> PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
>

Run on a single core with the debugger, -start_in_debugger, and get the
stack trace using 'where'.

  Thanks,

     Matt


> Kindly let me know.
>
> Venkatesh
>
> On Wed, May 27, 2015 at 2:07 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> Have you tried on a single process?
>>
>>   Thanks,
>>
>>     Matt
>>
>> On Tue, May 26, 2015 at 7:01 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi I am trying to solve my eigenproblem Ax=lambdaBx with a direct linear
>>> solver Pastix
>>>
>>> However I get this error (attached). It works without the pastix option.
>>>
>>> My matrices are in binary format.
>>>
>>> I gave the command : aprun -n 24 -N 24 ./ex7 -f1 a20t -f2 b20t -st_type
>>> sinvert -eps_nev 3 -log_summary -st_ksp_type preonly -st_pc_type lu
>>> -st_pc_factor_mat_solver_package pastix -ksp_monitor
>>>
>>> Kindly help me out.
>>>
>>> Venkatesh
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/80e18183/attachment.html>

From jed at jedbrown.org  Wed May 27 08:41:20 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 27 May 2015 15:41:20 +0200
Subject: [petsc-users] TS algo question
In-Reply-To: <CADOhEh4kFBGH37WCX_0OcrKPeg-R=VWH4+JxXLEH9R1Zg860pw@mail.gmail.com>
References: <CADOhEh4kFBGH37WCX_0OcrKPeg-R=VWH4+JxXLEH9R1Zg860pw@mail.gmail.com>
Message-ID: <87iobekuin.fsf@jedbrown.org>

Mark Adams <mfadams at lbl.gov> writes:

> I have setup the equations attached using TSSetIJacobian and
>  TSSetIFunction.
>
> This is intended to be a fully implicit solvers with some auxiliary
> variables.
>
> We want to verify the code with the current forward Euler solver.  

Mark, this is a DAE, so your "forward Euler" solver cannot literally be
forward Euler.  It can be a semi-implicit method with differential
variables propagated using forward Euler combined with solves at some
point.  If you want flexibility in treating terms implicitly and
explicitly, define both the RHSFunctian and IFunction.  Put those you
want to treat implicitly in the IFunction.  (And make the IJacobian the
Jacobian of the IFunction).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/9070c512/attachment.pgp>

From fande.kong at colorado.edu  Wed May 27 11:56:14 2015
From: fande.kong at colorado.edu (Fande Kong)
Date: Wed, 27 May 2015 10:56:14 -0600
Subject: [petsc-users] SNES Nonlinear solve did not converge due to
	DIVERGED_LINE_SEARCH
Message-ID: <CAN5Wd-KNrRqs3p73ZD8mjEsELjnW4Yq-H0gqhoDhYwgVxkRmfA@mail.gmail.com>

Hi all,

I was wondering if it is possible to let SNES  continue regardless of the
divergency of linear search, even though the residual only decreases a
little bit. In my test, if basic linear search is used, the snes converges
and  needs more iterations. But, BT does not work well although it
can guarantee a decrease in the residual.

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/f6f5f352/attachment.html>

From Xinya.Li at pnnl.gov  Wed May 27 11:58:34 2015
From: Xinya.Li at pnnl.gov (Li, Xinya)
Date: Wed, 27 May 2015 16:58:34 +0000
Subject: [petsc-users] Allocation errors
Message-ID: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>

Dear Sir/Madam,

I am trying to assembling a large sparse complex matrix for parallel computing. I need to pre-allocate for this sparse matrix A.

This is what I included in the code
***************************************************************************************************
// Allocate A
  ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen); CHKERRQ(ierr);
  ierr = MatSetFromOptions(A); CHKERRQ(ierr);
 ierr = MatSetUp(A); CHKERRQ(ierr);

  ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
  if (iteration > 0) {
    ierr = MatRetrieveValues(A); CHKERRQ(ierr);
  }
***************************************************************************************************

This is the error message
***************************************************************************************************
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
[0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729 Wed May 27 09:46:05 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++ --with-fortran-kernels=generic --with-cxx-dialect=C++11 --download-fblaslapack --download-mpich --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental
[0]PETSC ERROR: #1 MatGetOrdering() line 189 in /people/lixi729/petsc/src/mat/order/sorder.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 902 in /people/lixi729/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 306 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 418 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in /people/lixi729/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #7 SNESSolve() line 3743 in /people/lixi729/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #8 TSStep_Theta() line 194 in /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: #9 TSStep() line 2629 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #10 TSSolve() line 2748 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #11 simu() line 638 in simulation.C
[0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
****************************************************************************************************

Thank you very much. I really appreciate your help and time.
Regards
__________________________________________________
Xinya Li
Scientist
EED/Hydrology
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K9-33
Richland, WA  99352 USA
Tel:  509-372-6248
Fax: 509-372-6089
Xinya.Li at pnl.gov








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/6fc1bf88/attachment.html>

From knepley at gmail.com  Wed May 27 12:04:25 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 12:04:25 -0500
Subject: [petsc-users] Allocation errors
In-Reply-To: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
References: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
Message-ID: <CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>

On Wed, May 27, 2015 at 11:58 AM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:

>  Dear Sir/Madam,
>
>
>
> I am trying to assembling a large sparse complex matrix for parallel
> computing. I need to pre-allocate for this sparse matrix A.
>
>
>
> This is what I included in the code
>
>
> ***************************************************************************************************
>
> // Allocate A
>
>   ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
>
>   ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen);
> CHKERRQ(ierr);
>
>   ierr = MatSetFromOptions(A); CHKERRQ(ierr);
>
>  ierr = MatSetUp(A); CHKERRQ(ierr);
>
>
>
>   ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
>
>   if (iteration > 0) {
>
>     ierr = MatRetrieveValues(A); CHKERRQ(ierr);
>
>   }
>

Call MatAssemblyBeginEnd().

  Thanks,

    Matt


>
> ***************************************************************************************************
>
>
>
> This is the error message
>
>
> ***************************************************************************************************
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Object is in wrong state
>
> [0]PETSC ERROR: Not for unassembled matrix
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>
> [0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729
> Wed May 27 09:46:05 2015
>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
> --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++
> --with-fortran-kernels=generic --with-cxx-dialect=C++11
> --download-fblaslapack --download-mpich --download-superlu_dist
> --download-mumps --download-scalapack --download-parmetis --download-metis
> --download-elemental
>
> [0]PETSC ERROR: #1 MatGetOrdering() line 189 in
> /people/lixi729/petsc/src/mat/order/sorder.c
>
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in
> /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
>
> [0]PETSC ERROR: #3 PCSetUp() line 902 in
> /people/lixi729/petsc/src/ksp/pc/interface/precon.c
>
> [0]PETSC ERROR: #4 KSPSetUp() line 306 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #5 KSPSolve() line 418 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in
> /people/lixi729/petsc/src/snes/impls/ls/ls.c
>
> [0]PETSC ERROR: #7 SNESSolve() line 3743 in
> /people/lixi729/petsc/src/snes/interface/snes.c
>
> [0]PETSC ERROR: #8 TSStep_Theta() line 194 in
> /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
>
> [0]PETSC ERROR: #9 TSStep() line 2629 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #10 TSSolve() line 2748 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #11 simu() line 638 in simulation.C
>
> [0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
>
>
> ****************************************************************************************************
>
>
>
> Thank you very much. I really appreciate your help and time.
>
> Regards
>
> *__________________________________________________*
> *Xinya Li*
> Scientist
> EED/Hydrology
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K9-33
> Richland, WA  99352 USA
> Tel:  509-372-6248
> Fax: 509-372-6089
> *Xinya.Li at pnl.gov <Xinya.Li at pnl.gov>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/d8a43777/attachment.html>

From knepley at gmail.com  Wed May 27 12:05:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 12:05:18 -0500
Subject: [petsc-users] SNES Nonlinear solve did not converge due to
	DIVERGED_LINE_SEARCH
In-Reply-To: <CAN5Wd-KNrRqs3p73ZD8mjEsELjnW4Yq-H0gqhoDhYwgVxkRmfA@mail.gmail.com>
References: <CAN5Wd-KNrRqs3p73ZD8mjEsELjnW4Yq-H0gqhoDhYwgVxkRmfA@mail.gmail.com>
Message-ID: <CAMYG4GkHTEVS0rTHsB2-S6c3gqOR0z2sB1ZYN=7nYo+iwVQA+A@mail.gmail.com>

On Wed, May 27, 2015 at 11:56 AM, Fande Kong <fande.kong at colorado.edu>
wrote:

> Hi all,
>
> I was wondering if it is possible to let SNES  continue regardless of the
> divergency of linear search, even though the residual only decreases a
> little bit. In my test, if basic linear search is used, the snes converges
> and  needs more iterations. But, BT does not work well although it
> can guarantee a decrease in the residual.
>

http://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/SNES/SNESSetMaxNonlinearStepFailures.html

  Matt


> Thanks,
>
> Fande,
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/6e9d0319/attachment.html>

From hsahasra at purdue.edu  Wed May 27 12:22:46 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Wed, 27 May 2015 13:22:46 -0400
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAMYG4G=+gNFjhMr6xWjBRSS8NXeNa5jpE_b_uXRvwHnkAR_nHA@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
	<CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
	<CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>
	<CAMYG4G=+gNFjhMr6xWjBRSS8NXeNa5jpE_b_uXRvwHnkAR_nHA@mail.gmail.com>
Message-ID: <CAC0ZLSqsGmz821EFLqDbtng6qTkfb7iT2z+hvbjYp+g2++4uOg@mail.gmail.com>

Hi Matt,

Thanks for your comments. I see that SNESNRICHARDSON has changed a lot from
3.4.3 to 3.5.3. What do I need to do to make the code compatible with both
3.4.3 and 3.5.3?

Thanks,
Harshad

On Tue, May 26, 2015 at 4:25 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, May 26, 2015 at 3:07 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu
> > wrote:
>
>> Hi Matt,
>>
>> Actually I just checked, I don't need to re-init the preconditioner
>> LibMesh solver.
>>
>> The wrapper is interfaced to the mesh and makes it easier to assemble the
>> system matrices. It also keeps track of constraints. The wrapper function PetscNonlinearSolver::solve
>> updates the system vectors and matrices, then calls SNESSolve. Thus, I
>> need to call PetscNonlinearSolver::solve instead of SNESSolve in
>> SNESApplyNPC.
>>
>> Do you mean wrap it in SNESSHELL?
>>
>
> Yes, exactly.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Harshad
>>
>> On Tue, May 26, 2015 at 3:25 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <
>>> hsahasra at purdue.edu> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm using LibMesh for meshing a non-linear equation. I want to use
>>>> NRichardson non-linear solver to solve the equation. I have a Newton trust
>>>> region non-linear solver as a pre-conditioner. Both these SNES are created
>>>> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>>>>
>>>> I need to re-init the LibMesh NonlinearImplicitSystem each time the
>>>> preconditioner is applied. Also, I need the preconditioner to be solved via
>>>> the LibMesh wrapper instead of directly through SNESSolve. How can make it
>>>> work this way? Any help is appreciated!
>>>>
>>>
>>> I think the easiest way is to have them both look like SNES objects.
>>> Then you can just call SNESSetNPC() or use -snes_npc_side left
>>> -npc_snes_type newtonls
>>>
>>> Can you explain what the LibMesh wrapper is doing that you you need? We
>>> could wrap that object in a SNES, but it sounds
>>> simpler to me to unwrap it.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>> Harshad
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/472aa819/attachment.html>

From knepley at gmail.com  Wed May 27 12:25:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 12:25:58 -0500
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAC0ZLSqsGmz821EFLqDbtng6qTkfb7iT2z+hvbjYp+g2++4uOg@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
	<CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
	<CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>
	<CAMYG4G=+gNFjhMr6xWjBRSS8NXeNa5jpE_b_uXRvwHnkAR_nHA@mail.gmail.com>
	<CAC0ZLSqsGmz821EFLqDbtng6qTkfb7iT2z+hvbjYp+g2++4uOg@mail.gmail.com>
Message-ID: <CAMYG4Gng1ko1kC9q-KWBeLW4EC6VS4t8KQMZgmS6x5jXMCk_Kg@mail.gmail.com>

On Wed, May 27, 2015 at 12:22 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu>
wrote:

> Hi Matt,
>
> Thanks for your comments. I see that SNESNRICHARDSON has changed a lot
> from 3.4.3 to 3.5.3. What do I need to do to make the code compatible with
> both 3.4.3 and 3.5.3?
>

There are two issues:

  1) Making it compatible, which should be no problem since it is just a
SNES. We only changed the implementation.

  2) Comparing results. It may well be that 3.4.3 gives slightly different
results. I would have to look at the exact changes.

  Thanks,

     Matt


> Thanks,
> Harshad
>
> On Tue, May 26, 2015 at 4:25 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Tue, May 26, 2015 at 3:07 PM, Harshad Sahasrabudhe <
>> hsahasra at purdue.edu> wrote:
>>
>>> Hi Matt,
>>>
>>> Actually I just checked, I don't need to re-init the preconditioner
>>> LibMesh solver.
>>>
>>> The wrapper is interfaced to the mesh and makes it easier to assemble
>>> the system matrices. It also keeps track of constraints. The wrapper
>>> function PetscNonlinearSolver::solve updates the system vectors and
>>> matrices, then calls SNESSolve. Thus, I need to call PetscNonlinearSolver::solve
>>> instead of SNESSolve in SNESApplyNPC.
>>>
>>> Do you mean wrap it in SNESSHELL?
>>>
>>
>> Yes, exactly.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Harshad
>>>
>>> On Tue, May 26, 2015 at 3:25 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <
>>>> hsahasra at purdue.edu> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm using LibMesh for meshing a non-linear equation. I want to use
>>>>> NRichardson non-linear solver to solve the equation. I have a Newton trust
>>>>> region non-linear solver as a pre-conditioner. Both these SNES are created
>>>>> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>>>>>
>>>>> I need to re-init the LibMesh NonlinearImplicitSystem each time the
>>>>> preconditioner is applied. Also, I need the preconditioner to be solved via
>>>>> the LibMesh wrapper instead of directly through SNESSolve. How can make it
>>>>> work this way? Any help is appreciated!
>>>>>
>>>>
>>>> I think the easiest way is to have them both look like SNES objects.
>>>> Then you can just call SNESSetNPC() or use -snes_npc_side left
>>>> -npc_snes_type newtonls
>>>>
>>>> Can you explain what the LibMesh wrapper is doing that you you need? We
>>>> could wrap that object in a SNES, but it sounds
>>>> simpler to me to unwrap it.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Harshad
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/ef5399bc/attachment-0001.html>

From hsahasra at purdue.edu  Wed May 27 12:27:26 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Wed, 27 May 2015 13:27:26 -0400
Subject: [petsc-users] SNESNRICHARDSON with LibMesh
In-Reply-To: <CAMYG4Gng1ko1kC9q-KWBeLW4EC6VS4t8KQMZgmS6x5jXMCk_Kg@mail.gmail.com>
References: <CAC0ZLSoDjU7y3otXTAhVOp548_a32ipgU=5t=xaZUb+ifqfk-g@mail.gmail.com>
	<CAMYG4G=whU2E7P=DnAsyYdmHww-VPm1HAe1Eo-rfc7fi-PR2GQ@mail.gmail.com>
	<CAC0ZLSpkafpVDV3vytZQqNty1O5ZLDDA_x0GZf-y_xFbb-ULDA@mail.gmail.com>
	<CAMYG4G=+gNFjhMr6xWjBRSS8NXeNa5jpE_b_uXRvwHnkAR_nHA@mail.gmail.com>
	<CAC0ZLSqsGmz821EFLqDbtng6qTkfb7iT2z+hvbjYp+g2++4uOg@mail.gmail.com>
	<CAMYG4Gng1ko1kC9q-KWBeLW4EC6VS4t8KQMZgmS6x5jXMCk_Kg@mail.gmail.com>
Message-ID: <CAC0ZLSqZG=iu_dDv1yjUnwk-5pvD42y0AD_+jPfkBXnouWFL3A@mail.gmail.com>

Thanks, if the results are just slightly different with the same options,
then it's not a problem. I'll run my program using both to check.

On Wed, May 27, 2015 at 1:25 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 27, 2015 at 12:22 PM, Harshad Sahasrabudhe <
> hsahasra at purdue.edu> wrote:
>
>> Hi Matt,
>>
>> Thanks for your comments. I see that SNESNRICHARDSON has changed a lot
>> from 3.4.3 to 3.5.3. What do I need to do to make the code compatible with
>> both 3.4.3 and 3.5.3?
>>
>
> There are two issues:
>
>   1) Making it compatible, which should be no problem since it is just a
> SNES. We only changed the implementation.
>
>   2) Comparing results. It may well be that 3.4.3 gives slightly different
> results. I would have to look at the exact changes.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Harshad
>>
>> On Tue, May 26, 2015 at 4:25 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, May 26, 2015 at 3:07 PM, Harshad Sahasrabudhe <
>>> hsahasra at purdue.edu> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> Actually I just checked, I don't need to re-init the preconditioner
>>>> LibMesh solver.
>>>>
>>>> The wrapper is interfaced to the mesh and makes it easier to assemble
>>>> the system matrices. It also keeps track of constraints. The wrapper
>>>> function PetscNonlinearSolver::solve updates the system vectors and
>>>> matrices, then calls SNESSolve. Thus, I need to call PetscNonlinearSolver::solve
>>>> instead of SNESSolve in SNESApplyNPC.
>>>>
>>>> Do you mean wrap it in SNESSHELL?
>>>>
>>>
>>> Yes, exactly.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>> Harshad
>>>>
>>>> On Tue, May 26, 2015 at 3:25 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, May 26, 2015 at 12:42 PM, Harshad Sahasrabudhe <
>>>>> hsahasra at purdue.edu> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm using LibMesh for meshing a non-linear equation. I want to use
>>>>>> NRichardson non-linear solver to solve the equation. I have a Newton trust
>>>>>> region non-linear solver as a pre-conditioner. Both these SNES are created
>>>>>> via the LibMesh NonlinearImplicitSystem and PetscNonlinearSolver wrappers.
>>>>>>
>>>>>> I need to re-init the LibMesh NonlinearImplicitSystem each time the
>>>>>> preconditioner is applied. Also, I need the preconditioner to be solved via
>>>>>> the LibMesh wrapper instead of directly through SNESSolve. How can make it
>>>>>> work this way? Any help is appreciated!
>>>>>>
>>>>>
>>>>> I think the easiest way is to have them both look like SNES objects.
>>>>> Then you can just call SNESSetNPC() or use -snes_npc_side left
>>>>> -npc_snes_type newtonls
>>>>>
>>>>> Can you explain what the LibMesh wrapper is doing that you you need?
>>>>> We could wrap that object in a SNES, but it sounds
>>>>> simpler to me to unwrap it.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Harshad
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/87fd79b4/attachment.html>

From mfadams at lbl.gov  Wed May 27 12:32:26 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 28 May 2015 01:32:26 +0800
Subject: [petsc-users] TS algo question
In-Reply-To: <87iobekuin.fsf@jedbrown.org>
References: <CADOhEh4kFBGH37WCX_0OcrKPeg-R=VWH4+JxXLEH9R1Zg860pw@mail.gmail.com>
	<87iobekuin.fsf@jedbrown.org>
Message-ID: <CADOhEh479AgW3VL3py1bbc1+H5NRAFQvaWV+3Zr_mWDERpkhDQ@mail.gmail.com>

Thanks, So I think I can move the IJacobian terms for the time dependant
equations to a matrix and apply this matrix in the RHSFunction, and use
backward Euler, which will mainly compute the auxiliary terms in the
solve.  But then the auxiliary variables will be a function of the state at
the end of the time step, whereas the forward Euler (or what the existing
code does) is computed with the state at the beginning of the time step.
This may not be the best way to verify this code ... Thanks again.

On Wed, May 27, 2015 at 9:41 PM, Jed Brown <jed at jedbrown.org> wrote:

> Mark Adams <mfadams at lbl.gov> writes:
>
> > I have setup the equations attached using TSSetIJacobian and
> >  TSSetIFunction.
> >
> > This is intended to be a fully implicit solvers with some auxiliary
> > variables.
> >
> > We want to verify the code with the current forward Euler solver.
>
> Mark, this is a DAE, so your "forward Euler" solver cannot literally be
> forward Euler.  It can be a semi-implicit method with differential
> variables propagated using forward Euler combined with solves at some
> point.  If you want flexibility in treating terms implicitly and
> explicitly, define both the RHSFunctian and IFunction.  Put those you
> want to treat implicitly in the IFunction.  (And make the IJacobian the
> Jacobian of the IFunction).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/40153576/attachment.html>

From Xinya.Li at pnnl.gov  Wed May 27 12:39:24 2015
From: Xinya.Li at pnnl.gov (Li, Xinya)
Date: Wed, 27 May 2015 17:39:24 +0000
Subject: [petsc-users] Allocation errors
In-Reply-To: <CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>
References: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
	<CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>
Message-ID: <C7FD8207C10D14429D461DD4BDE868011A0E023B@EX10MBOX03.pnnl.gov>


The error messages were still the same.

In original code, I called  MatAssemblyBegin and  MatAssemblyEnd after I use MatSetValues to evaluate A.

From: Matthew Knepley [mailto:knepley at gmail.com]
Sent: Wednesday, May 27, 2015 10:04 AM
To: Li, Xinya
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Allocation errors

On Wed, May 27, 2015 at 11:58 AM, Li, Xinya <Xinya.Li at pnnl.gov<mailto:Xinya.Li at pnnl.gov>> wrote:
Dear Sir/Madam,

I am trying to assembling a large sparse complex matrix for parallel computing. I need to pre-allocate for this sparse matrix A.

This is what I included in the code
***************************************************************************************************
// Allocate A
  ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen); CHKERRQ(ierr);
  ierr = MatSetFromOptions(A); CHKERRQ(ierr);
 ierr = MatSetUp(A); CHKERRQ(ierr);

  ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
  if (iteration > 0) {
    ierr = MatRetrieveValues(A); CHKERRQ(ierr);
  }

Call MatAssemblyBeginEnd().

  Thanks,

    Matt

***************************************************************************************************

This is the error message
***************************************************************************************************
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
[0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729 Wed May 27 09:46:05 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++ --with-fortran-kernels=generic --with-cxx-dialect=C++11 --download-fblaslapack --download-mpich --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental
[0]PETSC ERROR: #1 MatGetOrdering() line 189 in /people/lixi729/petsc/src/mat/order/sorder.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 902 in /people/lixi729/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 306 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 418 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in /people/lixi729/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #7 SNESSolve() line 3743 in /people/lixi729/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #8 TSStep_Theta() line 194 in /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: #9 TSStep() line 2629 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #10 TSSolve() line 2748 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #11 simu() line 638 in simulation.C
[0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
****************************************************************************************************

Thank you very much. I really appreciate your help and time.
Regards
__________________________________________________
Xinya Li
Scientist
EED/Hydrology
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K9-33
Richland, WA  99352 USA
Tel:  509-372-6248<tel:509-372-6248>
Fax: 509-372-6089<tel:509-372-6089>
Xinya.Li at pnl.gov<mailto:Xinya.Li at pnl.gov>











--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/52c44fab/attachment-0001.html>

From knepley at gmail.com  Wed May 27 12:55:59 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 12:55:59 -0500
Subject: [petsc-users] Allocation errors
In-Reply-To: <C7FD8207C10D14429D461DD4BDE868011A0E023B@EX10MBOX03.pnnl.gov>
References: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
	<CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>
	<C7FD8207C10D14429D461DD4BDE868011A0E023B@EX10MBOX03.pnnl.gov>
Message-ID: <CAMYG4GkQFaeGx6qEGZ_cBEhsOgp6RO8=+VocqNnJ+GHjkjuAAg@mail.gmail.com>

On Wed, May 27, 2015 at 12:39 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:

>
>
> The error messages were still the same.
>
>
>
> In original code, I called  MatAssemblyBegin and  MatAssemblyEnd after I
> use MatSetValues to evaluate A.
>

That error message is given when a matrix you give PETSc is not assembled.
If the message persists, you have
called MatAssemblyBegin/End() on the wrong matrix. Maybe it would be easier
for you to use the debugger to
track down which matrix you need to assemble.

   Matt


>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* Wednesday, May 27, 2015 10:04 AM
> *To:* Li, Xinya
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Allocation errors
>
>
>
> On Wed, May 27, 2015 at 11:58 AM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>
> Dear Sir/Madam,
>
>
>
> I am trying to assembling a large sparse complex matrix for parallel
> computing. I need to pre-allocate for this sparse matrix A.
>
>
>
> This is what I included in the code
>
>
> ***************************************************************************************************
>
> // Allocate A
>
>   ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
>
>   ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen);
> CHKERRQ(ierr);
>
>   ierr = MatSetFromOptions(A); CHKERRQ(ierr);
>
>  ierr = MatSetUp(A); CHKERRQ(ierr);
>
>
>
>   ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
>
>   if (iteration > 0) {
>
>     ierr = MatRetrieveValues(A); CHKERRQ(ierr);
>
>   }
>
>
>
> Call MatAssemblyBeginEnd().
>
>
>
>   Thanks,
>
>
>
>     Matt
>
>
>
>
> ***************************************************************************************************
>
>
>
> This is the error message
>
>
> ***************************************************************************************************
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Object is in wrong state
>
> [0]PETSC ERROR: Not for unassembled matrix
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>
> [0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729
> Wed May 27 09:46:05 2015
>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
> --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++
> --with-fortran-kernels=generic --with-cxx-dialect=C++11
> --download-fblaslapack --download-mpich --download-superlu_dist
> --download-mumps --download-scalapack --download-parmetis --download-metis
> --download-elemental
>
> [0]PETSC ERROR: #1 MatGetOrdering() line 189 in
> /people/lixi729/petsc/src/mat/order/sorder.c
>
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in
> /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
>
> [0]PETSC ERROR: #3 PCSetUp() line 902 in
> /people/lixi729/petsc/src/ksp/pc/interface/precon.c
>
> [0]PETSC ERROR: #4 KSPSetUp() line 306 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #5 KSPSolve() line 418 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in
> /people/lixi729/petsc/src/snes/impls/ls/ls.c
>
> [0]PETSC ERROR: #7 SNESSolve() line 3743 in
> /people/lixi729/petsc/src/snes/interface/snes.c
>
> [0]PETSC ERROR: #8 TSStep_Theta() line 194 in
> /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
>
> [0]PETSC ERROR: #9 TSStep() line 2629 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #10 TSSolve() line 2748 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #11 simu() line 638 in simulation.C
>
> [0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
>
>
> ****************************************************************************************************
>
>
>
> Thank you very much. I really appreciate your help and time.
>
> Regards
>
> *__________________________________________________ *
> *Xinya Li*
> Scientist
> EED/Hydrology
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K9-33
> Richland, WA  99352 USA
> Tel:  509-372-6248
> Fax: 509-372-6089
> *Xinya.Li at pnl.gov <Xinya.Li at pnl.gov>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/22299e07/attachment.html>

From Xinya.Li at pnnl.gov  Wed May 27 13:16:17 2015
From: Xinya.Li at pnnl.gov (Li, Xinya)
Date: Wed, 27 May 2015 18:16:17 +0000
Subject: [petsc-users] Allocation errors
In-Reply-To: <CAMYG4GkQFaeGx6qEGZ_cBEhsOgp6RO8=+VocqNnJ+GHjkjuAAg@mail.gmail.com>
References: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
	<CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>
	<C7FD8207C10D14429D461DD4BDE868011A0E023B@EX10MBOX03.pnnl.gov>
	<CAMYG4GkQFaeGx6qEGZ_cBEhsOgp6RO8=+VocqNnJ+GHjkjuAAg@mail.gmail.com>
Message-ID: <C7FD8207C10D14429D461DD4BDE868011A0E0358@EX10MBOX03.pnnl.gov>

Thank you, Matthew.
Mat A is the only matrix I am working on in this function.
If I remove the  following lines from the code,  it will run smoothly.
// Allocate A
  ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen); CHKERRQ(ierr);
  ierr = MatSetFromOptions(A); CHKERRQ(ierr);
 ierr = MatSetUp(A); CHKERRQ(ierr);


From: Matthew Knepley [mailto:knepley at gmail.com]
Sent: Wednesday, May 27, 2015 10:56 AM
To: Li, Xinya
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Allocation errors

On Wed, May 27, 2015 at 12:39 PM, Li, Xinya <Xinya.Li at pnnl.gov<mailto:Xinya.Li at pnnl.gov>> wrote:

The error messages were still the same.

In original code, I called  MatAssemblyBegin and  MatAssemblyEnd after I use MatSetValues to evaluate A.

That error message is given when a matrix you give PETSc is not assembled. If the message persists, you have
called MatAssemblyBegin/End() on the wrong matrix. Maybe it would be easier for you to use the debugger to
track down which matrix you need to assemble.

   Matt


From: Matthew Knepley [mailto:knepley at gmail.com<mailto:knepley at gmail.com>]
Sent: Wednesday, May 27, 2015 10:04 AM
To: Li, Xinya
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Allocation errors

On Wed, May 27, 2015 at 11:58 AM, Li, Xinya <Xinya.Li at pnnl.gov<mailto:Xinya.Li at pnnl.gov>> wrote:
Dear Sir/Madam,

I am trying to assembling a large sparse complex matrix for parallel computing. I need to pre-allocate for this sparse matrix A.

This is what I included in the code
***************************************************************************************************
// Allocate A
  ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen); CHKERRQ(ierr);
  ierr = MatSetFromOptions(A); CHKERRQ(ierr);
 ierr = MatSetUp(A); CHKERRQ(ierr);

  ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
  if (iteration > 0) {
    ierr = MatRetrieveValues(A); CHKERRQ(ierr);
  }

Call MatAssemblyBeginEnd().

  Thanks,

    Matt

***************************************************************************************************

This is the error message
***************************************************************************************************
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
[0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729 Wed May 27 09:46:05 2015
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++ --with-fortran-kernels=generic --with-cxx-dialect=C++11 --download-fblaslapack --download-mpich --download-superlu_dist --download-mumps --download-scalapack --download-parmetis --download-metis --download-elemental
[0]PETSC ERROR: #1 MatGetOrdering() line 189 in /people/lixi729/petsc/src/mat/order/sorder.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 902 in /people/lixi729/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 306 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 418 in /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in /people/lixi729/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #7 SNESSolve() line 3743 in /people/lixi729/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #8 TSStep_Theta() line 194 in /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
[0]PETSC ERROR: #9 TSStep() line 2629 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #10 TSSolve() line 2748 in /people/lixi729/petsc/src/ts/interface/ts.c
[0]PETSC ERROR: #11 simu() line 638 in simulation.C
[0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
****************************************************************************************************

Thank you very much. I really appreciate your help and time.
Regards
__________________________________________________
Xinya Li
Scientist
EED/Hydrology
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K9-33
Richland, WA  99352 USA
Tel:  509-372-6248<tel:509-372-6248>
Fax: 509-372-6089<tel:509-372-6089>
Xinya.Li at pnl.gov<mailto:Xinya.Li at pnl.gov>











--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/611a1f8d/attachment-0001.html>

From mfadams at lbl.gov  Wed May 27 14:15:18 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 28 May 2015 03:15:18 +0800
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
Message-ID: <CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>

FYI, this does not seem to be fixed.

On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Looks like a bunch of these things are missing from
> lib/petsc/conf/bfort-petsc.txt  it should be listed under native
>
>   Barry
>
>
>
> > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > Does bfort have to be told that PCCompositeType is an enum?
> >
> >    Matt "bfort's biggest fan"
> >
> > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > It looks like fieldsplitf.c has an extra PetscToPointer.  type is
> already a pointer:
> >
> > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC
> pc,PCCompositeType *type, int *__ierr ){
> > *__ierr = PCFieldSplitGetType(
> >     (PC)PetscToPointer((pc) ),
> >     (PCCompositeType* )PetscToPointer((type) ));
> > }
> >
> > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    Likely need to run in the debugger to see what generates the error.
> Perhaps a missing ierr?
> >
> > Barry
> >
> >
> > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > >
> > > I declare
> > >
> > >   PCCompositeType::pcctype
> > >
> > > But PCFieldSplitGetType does not seem to like it?
> > >
> > > Mark
> > >
> > > [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > > [1]PETSC ERROR: Invalid pointer
> > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
> > > [1]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da
> GIT Date: 2015-04-16 11:32:06 -0500
> > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named
> MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
> > > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est
> --download-superlu=1 --download-mumps --download-scalapack
> --download-superlu_dist=1 --download-parmetis=1 --download-metis=1
> --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5
> --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
> > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in
> /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/a6972b78/attachment.html>

From knepley at gmail.com  Wed May 27 14:32:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 27 May 2015 14:32:35 -0500
Subject: [petsc-users] Allocation errors
In-Reply-To: <C7FD8207C10D14429D461DD4BDE868011A0E0358@EX10MBOX03.pnnl.gov>
References: <C7FD8207C10D14429D461DD4BDE868011A0E000D@EX10MBOX03.pnnl.gov>
	<CAMYG4GnsbVkjB0JbHhZF1W5f5jM0ka7oJzMumwhmk5M_khi5Rg@mail.gmail.com>
	<C7FD8207C10D14429D461DD4BDE868011A0E023B@EX10MBOX03.pnnl.gov>
	<CAMYG4GkQFaeGx6qEGZ_cBEhsOgp6RO8=+VocqNnJ+GHjkjuAAg@mail.gmail.com>
	<C7FD8207C10D14429D461DD4BDE868011A0E0358@EX10MBOX03.pnnl.gov>
Message-ID: <CAMYG4GkL6C9pyhiD5bHSyhKbZi0yQ2KVh5v8o65NhhEV59btOw@mail.gmail.com>

On Wed, May 27, 2015 at 1:16 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:

>  Thank you, Matthew.
>
> Mat A is the only matrix I am working on in this function.
>
> If I remove the  following lines from the code,  it will run smoothly.
>
> // Allocate A
>
>   ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
>
>   ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen);
> CHKERRQ(ierr);
>
>   ierr = MatSetFromOptions(A); CHKERRQ(ierr);
>
>  ierr = MatSetUp(A); CHKERRQ(ierr);
>

That error message will only happen if you do not assemble the matrix. Run
in the debugger, find
which matrix is unassembled, and assemble it.

  Thanks,

     Matt


>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* Wednesday, May 27, 2015 10:56 AM
> *To:* Li, Xinya
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Allocation errors
>
>
>
> On Wed, May 27, 2015 at 12:39 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>
>
>
> The error messages were still the same.
>
>
>
> In original code, I called  MatAssemblyBegin and  MatAssemblyEnd after I
> use MatSetValues to evaluate A.
>
>
>
> That error message is given when a matrix you give PETSc is not assembled.
> If the message persists, you have
>
> called MatAssemblyBegin/End() on the wrong matrix. Maybe it would be
> easier for you to use the debugger to
>
> track down which matrix you need to assemble.
>
>
>
>    Matt
>
>
>
>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* Wednesday, May 27, 2015 10:04 AM
> *To:* Li, Xinya
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Allocation errors
>
>
>
> On Wed, May 27, 2015 at 11:58 AM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>
> Dear Sir/Madam,
>
>
>
> I am trying to assembling a large sparse complex matrix for parallel
> computing. I need to pre-allocate for this sparse matrix A.
>
>
>
> This is what I included in the code
>
>
> ***************************************************************************************************
>
> // Allocate A
>
>   ierr = MatCreate(PETSC_COMM_WORLD,&A); CHKERRQ(ierr);
>
>   ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,4*ngen,4*ngen);
> CHKERRQ(ierr);
>
>   ierr = MatSetFromOptions(A); CHKERRQ(ierr);
>
>  ierr = MatSetUp(A); CHKERRQ(ierr);
>
>
>
>   ierr = MatGetOwnershipRange(A, &xstart, &xend); CHKERRQ(ierr);
>
>   if (iteration > 0) {
>
>     ierr = MatRetrieveValues(A); CHKERRQ(ierr);
>
>   }
>
>
>
> Call MatAssemblyBeginEnd().
>
>
>
>   Thanks,
>
>
>
>     Matt
>
>
>
>
> ***************************************************************************************************
>
>
>
> This is the error message
>
>
> ***************************************************************************************************
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> [0]PETSC ERROR: Object is in wrong state
>
> [0]PETSC ERROR: Not for unassembled matrix
>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>
> [0]PETSC ERROR: ./dynSim on a arch-complex named olympus.local by lixi729
> Wed May 27 09:46:05 2015
>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
> --with-cxx=g++ --with-scalar-type=complex --with-clanguage=C++
> --with-fortran-kernels=generic --with-cxx-dialect=C++11
> --download-fblaslapack --download-mpich --download-superlu_dist
> --download-mumps --download-scalapack --download-parmetis --download-metis
> --download-elemental
>
> [0]PETSC ERROR: #1 MatGetOrdering() line 189 in
> /people/lixi729/petsc/src/mat/order/sorder.c
>
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 196 in
> /people/lixi729/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
>
> [0]PETSC ERROR: #3 PCSetUp() line 902 in
> /people/lixi729/petsc/src/ksp/pc/interface/precon.c
>
> [0]PETSC ERROR: #4 KSPSetUp() line 306 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #5 KSPSolve() line 418 in
> /people/lixi729/petsc/src/ksp/ksp/interface/itfunc.c
>
> [0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 232 in
> /people/lixi729/petsc/src/snes/impls/ls/ls.c
>
> [0]PETSC ERROR: #7 SNESSolve() line 3743 in
> /people/lixi729/petsc/src/snes/interface/snes.c
>
> [0]PETSC ERROR: #8 TSStep_Theta() line 194 in
> /people/lixi729/petsc/src/ts/impls/implicit/theta/theta.c
>
> [0]PETSC ERROR: #9 TSStep() line 2629 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #10 TSSolve() line 2748 in
> /people/lixi729/petsc/src/ts/interface/ts.c
>
> [0]PETSC ERROR: #11 simu() line 638 in simulation.C
>
> [0]PETSC ERROR: #12 runSimulation() line 99 in dynSim.h
>
>
> ****************************************************************************************************
>
>
>
> Thank you very much. I really appreciate your help and time.
>
> Regards
>
> *__________________________________________________ *
> *Xinya Li*
> Scientist
> EED/Hydrology
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K9-33
> Richland, WA  99352 USA
> Tel:  509-372-6248
> Fax: 509-372-6089
> *Xinya.Li at pnl.gov <Xinya.Li at pnl.gov>*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/bb913457/attachment.html>

From bsmith at mcs.anl.gov  Wed May 27 15:03:06 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 27 May 2015 15:03:06 -0500
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
	<CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
Message-ID: <7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>


   Apparently all people with access to the PETSc repository (i.e. everyone in the world with internet access) have caught a rare disease that has caused their fingers to fall off, making them incapable of fixing this.

   I have added all the PETSc enums to this file in master and next. Make sure you run make allfortranstubs to regenerate the stubs.

   Barry



> On May 27, 2015, at 2:15 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> FYI, this does not seem to be fixed.
> 
> On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Looks like a bunch of these things are missing from lib/petsc/conf/bfort-petsc.txt  it should be listed under native
> 
>   Barry
> 
> 
> 
> > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > Does bfort have to be told that PCCompositeType is an enum?
> >
> >    Matt "bfort's biggest fan"
> >
> > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > It looks like fieldsplitf.c has an extra PetscToPointer.  type is already a pointer:
> >
> > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC pc,PCCompositeType *type, int *__ierr ){
> > *__ierr = PCFieldSplitGetType(
> >     (PC)PetscToPointer((pc) ),
> >     (PCCompositeType* )PetscToPointer((type) ));
> > }
> >
> > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    Likely need to run in the debugger to see what generates the error. Perhaps a missing ierr?
> >
> > Barry
> >
> >
> > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > >
> > > I declare
> > >
> > >   PCCompositeType::pcctype
> > >
> > > But PCFieldSplitGetType does not seem to like it?
> > >
> > > Mark
> > >
> > > [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > > [1]PETSC ERROR: Invalid pointer
> > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
> > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > > [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2702-g1f714da  GIT Date: 2015-04-16 11:32:06 -0500
> > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
> > > [1]PETSC ERROR: Configure options --download-hypre=1 --download-p4est --download-superlu=1 --download-mumps --download-scalapack --download-superlu_dist=1 --download-parmetis=1 --download-metis=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
> > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 


From mail2amneet at gmail.com  Wed May 27 22:40:37 2015
From: mail2amneet at gmail.com (Amneet Bhalla)
Date: Wed, 27 May 2015 23:40:37 -0400
Subject: [petsc-users] Handling singular matrix in KSP and ASM
Message-ID: <CAMETWJ1HhfcEaUO+HAiqE4gaHcfiP73WOPOz5yh+Wh6ydh7Sew@mail.gmail.com>

Hi Folks,

We are dealing with a singular matrix that arises from Stokes' equation
with Dirichlet velocity BCs. The way we are handling it is to pass
nullspace info to KSP, i.e, vector of {u,p} = {0,1}, and letting the matrix
to be singular. However, we are using ASM as the preconditioner and when
the subpc covers the entire domain (which can be the case in the
preconditioning step of our MG algorithm), LU decomposition fails. We are
thinking of pinning one of the pressure DOFs to zero explicitly, so that we
can avoid singular matrix for the LU decomposition. Is there another method
of doing it, which avoids explicit pinning down of an arbitrary pressure
DOF?

Thanks,
--Amneet

-- 
--Amneet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150527/a41ac6d0/attachment.html>

From bsmith at mcs.anl.gov  Wed May 27 22:52:11 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 27 May 2015 22:52:11 -0500
Subject: [petsc-users] Handling singular matrix in KSP and ASM
In-Reply-To: <CAMETWJ1HhfcEaUO+HAiqE4gaHcfiP73WOPOz5yh+Wh6ydh7Sew@mail.gmail.com>
References: <CAMETWJ1HhfcEaUO+HAiqE4gaHcfiP73WOPOz5yh+Wh6ydh7Sew@mail.gmail.com>
Message-ID: <53595CCF-8BEF-47BF-9B44-ABC0FB913513@mcs.anl.gov>


  The built in PETSc LU factorizations have special options to attempt to "shift" a singular matrix (by adding something to a zero pivot). These methods are ad hoc but sometimes perform satisfactory inside a preconditioner. See the manual page for PCLU in particular -pc_factor_shift_type <shifttype> - Sets shift type or PETSC_DECIDE for the default; use '-help' for a list of available types  See also http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot

  Barry


> On May 27, 2015, at 10:40 PM, Amneet Bhalla <mail2amneet at gmail.com> wrote:
> 
> Hi Folks,
> 
> We are dealing with a singular matrix that arises from Stokes' equation with Dirichlet velocity BCs. The way we are handling it is to pass nullspace info to KSP, i.e, vector of {u,p} = {0,1}, and letting the matrix to be singular. However, we are using ASM as the preconditioner and when the subpc covers the entire domain (which can be the case in the preconditioning step of our MG algorithm), LU decomposition fails. We are thinking of pinning one of the pressure DOFs to zero explicitly, so that we can avoid singular matrix for the LU decomposition. Is there another method of doing it, which avoids explicit pinning down of an arbitrary pressure DOF?
> 
> Thanks,
> --Amneet
> 
> -- 
> --Amneet 
> 
> 
> 


From hsahasra at purdue.edu  Thu May 28 00:34:08 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Thu, 28 May 2015 01:34:08 -0400
Subject: [petsc-users] Copy solution and function from one SNES to another
Message-ID: <CAC0ZLSorPWK7Q=6jrwiiW3FExcF-9yM3U9i_Dop6TD3CiTcQCA@mail.gmail.com>

Hi,

I'm trying to copy the solution and functions vecs from a Newton SNES to a
Shell SNES. I'm doing it like this:

  SNES newton_snes, shell_snes;
  Vec newton_function, newton_solution;
  Vec shell_function, shell_solution;
  .
  .
  Newton SNES initialized and solved here. Shell SNES initialized.
  .
  .
  SNESGetFunction(newton_snes, &newton_function, 0, 0);
  SNESGetFunction(shell_snes, &shell_function, 0, 0);
  VecDestroy(&shell_function);
  VecDuplicate(newton_function, &shell_function);

  SNESGetSolution(newton_snes, &newton_solution);
  SNESGetSolution(shell_snes, &shell_solution);
  VecDestroy(&shell_solution);
  VecDuplicate(newton_solution, &shell_solution);

But I get this error on first VecDuplicate line:

[0]PETSC ERROR: VecDuplicate() line 511 in
/home/hsahasra/NEMO5/libs/petsc/build-real/src/vec/vec/interface/vector.c
Null Object: Parameter # 1

Is there any other better way to do this?

Thanks,
Harshad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/fb438457/attachment.html>

From riseth at maths.ox.ac.uk  Thu May 28 03:15:43 2015
From: riseth at maths.ox.ac.uk (=?UTF-8?Q?Asbj=C3=B8rn_Nilsen_Riseth?=)
Date: Thu, 28 May 2015 08:15:43 +0000
Subject: [petsc-users] SNESVINEWTONRSLS: definition of active set?
Message-ID: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>

Dear PETSc developers,

Is the active set in NewtonRSLS defined differently from the reference* you
give in the documentation on purpose?
The reference defines the active set as:
x_i = 0 and F_i > 0,
whilst the PETSc code defines it as x_i = 0 and F_i >= 0 (vi.c: 356) :

!((PetscRealPart(x[i]) > PetscRealPart(xl[i]) + 1.e-8 ||
(PetscRealPart(f[i]) < 0.0)

So PETSc freezes the variables if f[i] == 0.

I've been using the Newton RSLS method to ensure positivity in a subsurface
flow problem I'm working on. My solution stays almost constant for two
timesteps (seemingly independent of the size of the timestep), before it
goes towards the expected solution.
>From my initial conditions, certain variables are frozen because x_i = 0
and f[i] = 0, and I was wondering if that could be the cause of my issue.


*:
- T. S. Munson, and S. Benson. Flexible Complementarity Solvers for
Large-Scale Applications, Optimization Methods and Software, 21 (2006).


Regards,
Ozzy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/75731575/attachment.html>

From knepley at gmail.com  Thu May 28 06:20:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 28 May 2015 06:20:45 -0500
Subject: [petsc-users] Handling singular matrix in KSP and ASM
In-Reply-To: <53595CCF-8BEF-47BF-9B44-ABC0FB913513@mcs.anl.gov>
References: <CAMETWJ1HhfcEaUO+HAiqE4gaHcfiP73WOPOz5yh+Wh6ydh7Sew@mail.gmail.com>
	<53595CCF-8BEF-47BF-9B44-ABC0FB913513@mcs.anl.gov>
Message-ID: <CAMYG4G=QMAs4nZDKL1ak98PDjp+hjnag24b9TQdXRiCAweRJcg@mail.gmail.com>

On Wed, May 27, 2015 at 10:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   The built in PETSc LU factorizations have special options to attempt to
> "shift" a singular matrix (by adding something to a zero pivot). These
> methods are ad hoc but sometimes perform satisfactory inside a
> preconditioner. See the manual page for PCLU in particular
> -pc_factor_shift_type <shifttype> - Sets shift type or PETSC_DECIDE for the
> default; use '-help' for a list of available types  See also
> http://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot


The other option is to use -pc_type svd for the coarse grid.

  Matt


>
>   Barry
>
>
> > On May 27, 2015, at 10:40 PM, Amneet Bhalla <mail2amneet at gmail.com>
> wrote:
> >
> > Hi Folks,
> >
> > We are dealing with a singular matrix that arises from Stokes' equation
> with Dirichlet velocity BCs. The way we are handling it is to pass
> nullspace info to KSP, i.e, vector of {u,p} = {0,1}, and letting the matrix
> to be singular. However, we are using ASM as the preconditioner and when
> the subpc covers the entire domain (which can be the case in the
> preconditioning step of our MG algorithm), LU decomposition fails. We are
> thinking of pinning one of the pressure DOFs to zero explicitly, so that we
> can avoid singular matrix for the LU decomposition. Is there another method
> of doing it, which avoids explicit pinning down of an arbitrary pressure
> DOF?
> >
> > Thanks,
> > --Amneet
> >
> > --
> > --Amneet
> >
> >
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/3421ff05/attachment.html>

From knepley at gmail.com  Thu May 28 06:25:05 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 28 May 2015 06:25:05 -0500
Subject: [petsc-users] Copy solution and function from one SNES to
	another
In-Reply-To: <CAC0ZLSorPWK7Q=6jrwiiW3FExcF-9yM3U9i_Dop6TD3CiTcQCA@mail.gmail.com>
References: <CAC0ZLSorPWK7Q=6jrwiiW3FExcF-9yM3U9i_Dop6TD3CiTcQCA@mail.gmail.com>
Message-ID: <CAMYG4GmxC9f0Moyvpkbzm_pL=W8y02VFXL34umNNHbO2sFuh1Q@mail.gmail.com>

On Thu, May 28, 2015 at 12:34 AM, Harshad Sahasrabudhe <hsahasra at purdue.edu>
wrote:

> Hi,
>
> I'm trying to copy the solution and functions vecs from a Newton SNES to a
> Shell SNES. I'm doing it like this:
>
>   SNES newton_snes, shell_snes;
>   Vec newton_function, newton_solution;
>   Vec shell_function, shell_solution;
>   .
>   .
>   Newton SNES initialized and solved here. Shell SNES initialized.
>   .
>   .
>   SNESGetFunction(newton_snes, &newton_function, 0, 0);
>   SNESGetFunction(shell_snes, &shell_function, 0, 0);
>   VecDestroy(&shell_function);
>   VecDuplicate(newton_function, &shell_function);
>

If you want to use the same vector

  SNESGetFunction(newton, &f)
  SNESSetFunction(shell, f)

otherwise for a copy

  SNESGetFunction(newton, &f)
  VecDuplicate(f, &sf)
  SNESSetFunction(shell, sf)
  VecDestroy(&sf)

  Thanks,

     Matt



>   SNESGetSolution(newton_snes, &newton_solution);
>   SNESGetSolution(shell_snes, &shell_solution);
>   VecDestroy(&shell_solution);
>   VecDuplicate(newton_solution, &shell_solution);
>
> But I get this error on first VecDuplicate line:
>
> [0]PETSC ERROR: VecDuplicate() line 511 in
> /home/hsahasra/NEMO5/libs/petsc/build-real/src/vec/vec/interface/vector.c
> Null Object: Parameter # 1
>
> Is there any other better way to do this?
>
> Thanks,
> Harshad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/64b38512/attachment-0001.html>

From elias.karabelas at medunigraz.at  Thu May 28 10:47:17 2015
From: elias.karabelas at medunigraz.at (Elias Karabelas)
Date: Thu, 28 May 2015 17:47:17 +0200
Subject: [petsc-users] Block Preconditioning
Message-ID: <55673885.4020802@medunigraz.at>

Dear Members,

I want to solve a Block System arising from the Discretization of a 
stabilized Finite Element Formulation of the Stokes System.

I have the following Block Structure

A     -B^T
B       C

The Preconditioner I intend to use is a block preconditioner of the Form

A    -B^T
         S

where S is an approximation of the Schur Complement. For applying the 
inverse of the schur complement I want to use a Stabilized Least Squares 
Commutator in the form

S^-1 = (B diag(Q)^-1 B^T + C_1)^-1 (B diag(Q)^-1 A diag(Q)^-1 B^T + C_2) 
(B diag(Q)^-1 B^T + C_1)^-1

where Q is the mass matrix and C_1 and C_2 are some additional 
stabilization matrices.

I got from the Manual, that I can use the PCFieldSplit preconditioner 
for generating the general Block preconditioner as indicated above. And 
I also found that I can define some arbitrary PC with PCSHELL. My 
question is, if it is possible to use PCSHELL to define the action of 
S^-1 as indicated above.

Kind Regards
Elias Karabelas

-- 
Elias Karabelas, Ph.D.

Medical University of Graz
Institute of Biophysics
Harrachgasse 21/IV
8010 Graz, Austria

Phone: +43 316 380 7759
Email: elias.karabelas at medunigraz.at
Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas


From knepley at gmail.com  Thu May 28 11:02:13 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 28 May 2015 11:02:13 -0500
Subject: [petsc-users] Block Preconditioning
In-Reply-To: <55673885.4020802@medunigraz.at>
References: <55673885.4020802@medunigraz.at>
Message-ID: <CAMYG4Gk-QAbk=95jpjH_GOBmfaFLOOfEamrA3TmXcuTjKs5Lcw@mail.gmail.com>

On Thu, May 28, 2015 at 10:47 AM, Elias Karabelas <
elias.karabelas at medunigraz.at> wrote:

> Dear Members,
>
> I want to solve a Block System arising from the Discretization of a
> stabilized Finite Element Formulation of the Stokes System.
>
> I have the following Block Structure
>
> A     -B^T
> B       C
>
> The Preconditioner I intend to use is a block preconditioner of the Form
>
> A    -B^T
>         S
>
> where S is an approximation of the Schur Complement. For applying the
> inverse of the schur complement I want to use a Stabilized Least Squares
> Commutator in the form
>
> S^-1 = (B diag(Q)^-1 B^T + C_1)^-1 (B diag(Q)^-1 A diag(Q)^-1 B^T + C_2)
> (B diag(Q)^-1 B^T + C_1)^-1
>
> where Q is the mass matrix and C_1 and C_2 are some additional
> stabilization matrices.
>
> I got from the Manual, that I can use the PCFieldSplit preconditioner for
> generating the general Block preconditioner as indicated above. And I also
> found that I can define some arbitrary PC with PCSHELL. My question is, if
> it is possible to use PCSHELL to define the action of S^-1 as indicated
> above.
>

1) Use FieldSplit is the right PC to start with. Make sure you can do
something simple like

  A -B^T
      C + B diag(A)^{-1} B^T

with it before we do the more complicated thing.

2) You will want to implement a PC for the (1,1) block. You can use a
PCSHELL, which is simpler to setup, but
    that means you will have to manually pull out the FieldSplit KSP and
set it. If instead you define your own
    PC implementation, its more boilerplate code, but you could specify
this PC from the command line without
    any FieldSplit specific code in your application.

3) Your PC will get two matrices, the MatSchurComplement, and the
preconditioning matrix. If you set Q as the
     preconditioning matrix, or really if you set

    A   0
    0   Q

as the global preconditioning matrix, then the subsolve for (1,1) will get
the Schur Complement and Q, and I think
that is enough to build your Stabilized LSC PC.

Let me know if this makes sense to you.

  Thanks,

    Matt

Kind Regards
> Elias Karabelas
>
> --
> Elias Karabelas, Ph.D.
>
> Medical University of Graz
> Institute of Biophysics
> Harrachgasse 21/IV
> 8010 Graz, Austria
>
> Phone: +43 316 380 7759
> Email: elias.karabelas at medunigraz.at
> Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/32687b7b/attachment.html>

From elias.karabelas at medunigraz.at  Thu May 28 11:09:13 2015
From: elias.karabelas at medunigraz.at (Elias Karabelas)
Date: Thu, 28 May 2015 18:09:13 +0200
Subject: [petsc-users] Block Preconditioning
In-Reply-To: <CAMYG4Gk-QAbk=95jpjH_GOBmfaFLOOfEamrA3TmXcuTjKs5Lcw@mail.gmail.com>
References: <55673885.4020802@medunigraz.at>
	<CAMYG4Gk-QAbk=95jpjH_GOBmfaFLOOfEamrA3TmXcuTjKs5Lcw@mail.gmail.com>
Message-ID: <55673DA9.2010601@medunigraz.at>

Dear Matt,

sorry but I think I haven't understood points 2 and 3 you mentioned. 
Could you be a bit more specific?

Thanks
Elias

On 28.05.2015 18:02, Matthew Knepley wrote:
> On Thu, May 28, 2015 at 10:47 AM, Elias Karabelas 
> <elias.karabelas at medunigraz.at <mailto:elias.karabelas at medunigraz.at>> 
> wrote:
>
>     Dear Members,
>
>     I want to solve a Block System arising from the Discretization of
>     a stabilized Finite Element Formulation of the Stokes System.
>
>     I have the following Block Structure
>
>     A     -B^T
>     B       C
>
>     The Preconditioner I intend to use is a block preconditioner of
>     the Form
>
>     A    -B^T
>             S
>
>     where S is an approximation of the Schur Complement. For applying
>     the inverse of the schur complement I want to use a Stabilized
>     Least Squares Commutator in the form
>
>     S^-1 = (B diag(Q)^-1 B^T + C_1)^-1 (B diag(Q)^-1 A diag(Q)^-1 B^T
>     + C_2) (B diag(Q)^-1 B^T + C_1)^-1
>
>     where Q is the mass matrix and C_1 and C_2 are some additional
>     stabilization matrices.
>
>     I got from the Manual, that I can use the PCFieldSplit
>     preconditioner for generating the general Block preconditioner as
>     indicated above. And I also found that I can define some arbitrary
>     PC with PCSHELL. My question is, if it is possible to use PCSHELL
>     to define the action of S^-1 as indicated above.
>
>
> 1) Use FieldSplit is the right PC to start with. Make sure you can do 
> something simple like
>
>   A -B^T
>       C + B diag(A)^{-1} B^T
>
> with it before we do the more complicated thing.
>
> 2) You will want to implement a PC for the (1,1) block. You can use a 
> PCSHELL, which is simpler to setup, but
>     that means you will have to manually pull out the FieldSplit KSP 
> and set it. If instead you define your own
>     PC implementation, its more boilerplate code, but you could 
> specify this PC from the command line without
>     any FieldSplit specific code in your application.
>
> 3) Your PC will get two matrices, the MatSchurComplement, and the 
> preconditioning matrix. If you set Q as the
>      preconditioning matrix, or really if you set
>
>     A   0
>     0   Q
> as the global preconditioning matrix, then the subsolve for (1,1) will 
> get the Schur Complement and Q, and I think
> that is enough to build your Stabilized LSC PC.
>
> Let me know if this makes sense to you.
>
>   Thanks,
>
>     Matt
>
>     Kind Regards
>     Elias Karabelas
>
>     -- 
>     Elias Karabelas, Ph.D.
>
>     Medical University of Graz
>     Institute of Biophysics
>     Harrachgasse 21/IV
>     8010 Graz, Austria
>
>     Phone: +43 316 380 7759 <tel:%2B43%20316%20380%207759>
>     Email: elias.karabelas at medunigraz.at
>     <mailto:elias.karabelas at medunigraz.at>
>     Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-- 
Elias Karabelas, Ph.D.

Medical University of Graz
Institute of Biophysics
Harrachgasse 21/IV
8010 Graz, Austria

Phone: +43 316 380 7759
Email: elias.karabelas at medunigraz.at
Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150528/954c25a9/attachment.html>

From bsmith at mcs.anl.gov  Thu May 28 14:26:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 28 May 2015 14:26:23 -0500
Subject: [petsc-users] SNESVINEWTONRSLS: definition of active set?
In-Reply-To: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>
References: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>
Message-ID: <5ABC2C8A-1BE6-489C-BACD-E8D780CD205A@mcs.anl.gov>


  Ozzy,

     I cannot say why it is implemented as >=  (could be in error). Just try changing the PETSc code (just run make gnumake in the PETSc directory after you change the source to update the library) and see how it affects your code run.

   Barry

> On May 28, 2015, at 3:15 AM, Asbj?rn Nilsen Riseth <riseth at maths.ox.ac.uk> wrote:
> 
> Dear PETSc developers,
> 
> Is the active set in NewtonRSLS defined differently from the reference* you give in the documentation on purpose?
> The reference defines the active set as:
> x_i = 0 and F_i > 0,
> whilst the PETSc code defines it as x_i = 0 and F_i >= 0 (vi.c: 356) :
> !((PetscRealPart(x[i]) > PetscRealPart(xl[i]) + 1.e-8 || (PetscRealPart(f[i]) < 0.0)
> So PETSc freezes the variables if f[i] == 0.
> 
> I've been using the Newton RSLS method to ensure positivity in a subsurface flow problem I'm working on. My solution stays almost constant for two timesteps (seemingly independent of the size of the timestep), before it goes towards the expected solution.
> From my initial conditions, certain variables are frozen because x_i = 0 and f[i] = 0, and I was wondering if that could be the cause of my issue.
> 
> 
> *:
> - T. S. Munson, and S. Benson. Flexible Complementarity Solvers for Large-Scale Applications, Optimization Methods and Software, 21 (2006).
> 
> 
> Regards,
> Ozzy


From venkateshgk.j at gmail.com  Fri May 29 02:56:18 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 29 May 2015 13:26:18 +0530
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CAGCphBuh9Nco8SvQhfsLC9AYnt9Uj_Nfyy_muaSxJRGhspdsUg@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CAGCphBuh9Nco8SvQhfsLC9AYnt9Uj_Nfyy_muaSxJRGhspdsUg@mail.gmail.com>
Message-ID: <CABzT7hM_VJcwWwkGkZ1A1-O0EM2-ShZjuWjE5b93nqo=_sWWGA@mail.gmail.com>

On Tue, May 26, 2015 at 9:02 PM, Hong <hzhang at mcs.anl.gov> wrote:

> 'A serial job in MATLAB for the same matrices takes < 60GB. '
> Can you run this case in serial? If so, try petsc, superlu or mumps to
> make sure the matrix is non-singular.
>
B matrix is singular but I get my result in Petsc and Mumps for small
matrices.

>
> Both mumps and superlu_dist show crash in MatFactorNumeric(). Mumps gives
> error
> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization phase:
> Cannot allocate required memory 65710 megabytes.
>
> Does your code work for smaller problems?
>
Yes code works for small problems

> Try using more processors?
>
> Why use such huge '-mat_mumps_icntl_14 200000' (percentage of estimated
> workspace increase)? The default is 20. Try 40?
>
> Superlu_dist usually uses less memory than mumps, but it also crashes. I
> guess something wrong with your matrix. Is it singular?
>
The B matrix is singular. So Super Lu cant be used is it ?


> Run superlu_dist on a slightly smaller matrix with option
> '-mat_superlu_dist_statprint' which displays memory usage info., e.g.,
>
Ok I will do that and check.

> petsc/src/ksp/ksp/examples/tutorials (maint)
> $ mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package
> superlu_dist -mat_superlu_dist_statprint
> Nonzeros in L       560
> Nonzeros in U       560
> nonzeros in L+U     1064
> nonzeros in LSUB    248
> NUMfact space (MB) sum(procs):  L\U 0.01 all 0.05
> Total highmark (MB):  All 0.05 Avg 0.02 Max 0.02
> EQUIL time             0.00
> ROWPERM time           0.00
> COLPERM time           0.00
> SYMBFACT time          0.00
> DISTRIBUTE time        0.00
> FACTOR time            0.00
> Factor flops 1.181000e+04 Mflops     4.80
> SOLVE time             0.00
> SOLVE time             0.00
> Solve flops 2.194000e+03 Mflops     4.43
> SOLVE time             0.00
> Solve flops 2.194000e+03 Mflops     5.14
> Norm of error 1.18018e-15 iterations 1
>
> Hong
>
>
> On Tue, May 26, 2015 at 9:03 AM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> I posted a while ago in MUMPS forums but no one seems to reply.
>>
>> I am solving a large generalized Eigenvalue problem.
>>
>> I am getting the following error which is attached, after giving the
>> command:
>>
>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64 -hosts
>> compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2 b72t
>> -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>> 200000
>>
>> IT IS impossible to allocate so much memory per processor.. it is asking
>> like around 70 GB per processor.
>>
>> A serial job in MATLAB for the same matrices takes < 60GB.
>>
>> After trying out superLU_dist, I have attached the error there also
>> (segmentation error).
>>
>> Kindly help me.
>>
>> Venkatesh
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/46ab3c24/attachment.html>

From riseth at maths.ox.ac.uk  Fri May 29 04:19:14 2015
From: riseth at maths.ox.ac.uk (=?UTF-8?Q?Asbj=C3=B8rn_Nilsen_Riseth?=)
Date: Fri, 29 May 2015 09:19:14 +0000
Subject: [petsc-users] SNESVINEWTONRSLS: definition of active set?
In-Reply-To: <5ABC2C8A-1BE6-489C-BACD-E8D780CD205A@mcs.anl.gov>
References: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>
	<5ABC2C8A-1BE6-489C-BACD-E8D780CD205A@mcs.anl.gov>
Message-ID: <CALFwutOr9cGq=KjDCwniyBEBDTh9LMuuZPEqb0r7Vbn=jLa9jA@mail.gmail.com>

Barry,

I changed the code, but it only makes a difference at the order of 1e-10 -
so that's not the cause. I've attached (if that's appropriate?) the patch
in case anyone is interested.

Investigating the function values, I see that the first Newton step goes
towards the expected solution for my function values. Then it shoots back
close to the initial conditions.
At the time the solver hits tolerance for the inactive set; the function
value is 100-1000 at some of the active set indices.
Reducing the timestep by an order of magnitude shows the same behavior for
the first two timesteps.

Maybe VI is not the right approach. The company I work with seem to just be
projecting negative values.

I'll investigate further.

Ozzy

On Thu, 28 May 2015 at 20:26 Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Ozzy,
>
>      I cannot say why it is implemented as >=  (could be in error). Just
> try changing the PETSc code (just run make gnumake in the PETSc directory
> after you change the source to update the library) and see how it affects
> your code run.
>
>    Barry
>
> > On May 28, 2015, at 3:15 AM, Asbj?rn Nilsen Riseth <
> riseth at maths.ox.ac.uk> wrote:
> >
> > Dear PETSc developers,
> >
> > Is the active set in NewtonRSLS defined differently from the reference*
> you give in the documentation on purpose?
> > The reference defines the active set as:
> > x_i = 0 and F_i > 0,
> > whilst the PETSc code defines it as x_i = 0 and F_i >= 0 (vi.c: 356) :
> > !((PetscRealPart(x[i]) > PetscRealPart(xl[i]) + 1.e-8 ||
> (PetscRealPart(f[i]) < 0.0)
> > So PETSc freezes the variables if f[i] == 0.
> >
> > I've been using the Newton RSLS method to ensure positivity in a
> subsurface flow problem I'm working on. My solution stays almost constant
> for two timesteps (seemingly independent of the size of the timestep),
> before it goes towards the expected solution.
> > From my initial conditions, certain variables are frozen because x_i = 0
> and f[i] = 0, and I was wondering if that could be the cause of my issue.
> >
> >
> > *:
> > - T. S. Munson, and S. Benson. Flexible Complementarity Solvers for
> Large-Scale Applications, Optimization Methods and Software, 21 (2006).
> >
> >
> > Regards,
> > Ozzy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/8e4674cf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Define-active-and-inactive-sets-correctly.patch
Type: text/x-patch
Size: 4785 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/8e4674cf/attachment.bin>

From karabelaselias at gmail.com  Fri May 29 05:09:17 2015
From: karabelaselias at gmail.com (Elias Karabelas)
Date: Fri, 29 May 2015 12:09:17 +0200
Subject: [petsc-users] Block Preconditioning
Message-ID: <CAM1BeEkpeW5JqYuuZATfEFm_b=JwCrbCairrZvtsQGt+t7B3dw@mail.gmail.com>

Dear Matt,

I tried out what you said. I used the following example I found with google

https://raw.githubusercontent.com/petsc/petsc/5edff71f342f05166073de0ae93226f0997d7fe9/src/snes/examples/tutorials/ex70.c

So now I have some questions concerning that


a) When I use pc_fieldsplit_precondition_schur selfp I get slower
convergence than with the user-provided Schur-Complement approximation in
this example code. However from what I saw this should coincide in this
case?

b) I get a little confused with the settings of the inner and outer solvers
in fieldsplit. So if I understood it correctly
"-pc_fieldsplit_schur_fact_type upper" uses

K    -B^T
0     S

as preconditioner. Where K and S are inverted with the help of a KSP. So
what I don't get is how to tweak the inner solvers correctly. For Example,
In my own Implementation of preconditioned GMRES I also use the same
structure as above as preconditioner and I take K^-1 to be approximated by
the application of an AMG (in my case BoomerAMG) . Further I take S = B
diag(K)^-1 B^T and again use an AMG to invert S. This results in my case in
a good preconditioner.

I tried now the same with this example and used

-fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type hypre
-fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type hypre

However I get really slow convergence of the outer GMRES method.

Can someone give me some insight into this?

Best regards
Elias


On 28.05.2015 18:02, Matthew Knepley wrote:

On Thu, May 28, 2015 at 10:47 AM, Elias Karabelas <
elias.karabelas at medunigraz.at> wrote:

> Dear Members,
>
> I want to solve a Block System arising from the Discretization of a
> stabilized Finite Element Formulation of the Stokes System.
>
> I have the following Block Structure
>
> A     -B^T
> B       C
>
> The Preconditioner I intend to use is a block preconditioner of the Form
>
> A    -B^T
>         S
>
> where S is an approximation of the Schur Complement. For applying the
> inverse of the schur complement I want to use a Stabilized Least Squares
> Commutator in the form
>
> S^-1 = (B diag(Q)^-1 B^T + C_1)^-1 (B diag(Q)^-1 A diag(Q)^-1 B^T + C_2)
> (B diag(Q)^-1 B^T + C_1)^-1
>
> where Q is the mass matrix and C_1 and C_2 are some additional
> stabilization matrices.
>
> I got from the Manual, that I can use the PCFieldSplit preconditioner for
> generating the general Block preconditioner as indicated above. And I also
> found that I can define some arbitrary PC with PCSHELL. My question is, if
> it is possible to use PCSHELL to define the action of S^-1 as indicated
> above.
>

1) Use FieldSplit is the right PC to start with. Make sure you can do
something simple like

  A -B^T
      C + B diag(A)^{-1} B^T

with it before we do the more complicated thing.

2) You will want to implement a PC for the (1,1) block. You can use a
PCSHELL, which is simpler to setup, but
    that means you will have to manually pull out the FieldSplit KSP and
set it. If instead you define your own
    PC implementation, its more boilerplate code, but you could specify
this PC from the command line without
    any FieldSplit specific code in your application.

3) Your PC will get two matrices, the MatSchurComplement, and the
preconditioning matrix. If you set Q as the
     preconditioning matrix, or really if you set

    A   0
    0   Q

as the global preconditioning matrix, then the subsolve for (1,1) will get
the Schur Complement and Q, and I think
that is enough to build your Stabilized LSC PC.

Let me know if this makes sense to you.

  Thanks,

    Matt

Kind Regards
> Elias Karabelas
>
> --
> Elias Karabelas, Ph.D.
>
> Medical University of Graz
> Institute of Biophysics
> Harrachgasse 21/IV
> 8010 Graz, Austria
>
> Phone: +43 316 380 7759
> Email: elias.karabelas at medunigraz.at
> Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/4fec5d7a/attachment-0001.html>

From knepley at gmail.com  Fri May 29 06:18:54 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 29 May 2015 06:18:54 -0500
Subject: [petsc-users] Block Preconditioning
In-Reply-To: <CAM1BeEkpeW5JqYuuZATfEFm_b=JwCrbCairrZvtsQGt+t7B3dw@mail.gmail.com>
References: <CAM1BeEkpeW5JqYuuZATfEFm_b=JwCrbCairrZvtsQGt+t7B3dw@mail.gmail.com>
Message-ID: <CAMYG4G=zSBC3Bz5-ODUijgcaDqoRq9N_8Ry2E_1ZyKVM2E9jkg@mail.gmail.com>

On Fri, May 29, 2015 at 5:09 AM, Elias Karabelas <karabelaselias at gmail.com>
wrote:

> Dear Matt,
>
> I tried out what you said. I used the following example I found with google
>
>
> https://raw.githubusercontent.com/petsc/petsc/5edff71f342f05166073de0ae93226f0997d7fe9/src/snes/examples/tutorials/ex70.c
>

Please do not start with this one. It exhibits the manual approach, which
is necessary sometimes, but
definitely less elegant than the command line.


> So now I have some questions concerning that
>
>
> a) When I use pc_fieldsplit_precondition_schur selfp I get slower
> convergence than with the user-provided Schur-Complement approximation in
> this example code. However from what I saw this should coincide in this
> case?
>

For the Stokes Equation selfp gives a Schur preconditioner that is not
spectrally equivalent:

  B^T diag(A)^{-1} B \approx \Delta    whereas   S \approx I

so we generally use something that looks like that mass matrix. I think
that is what is in the code.


> b) I get a little confused with the settings of the inner and outer
> solvers in fieldsplit. So if I understood it correctly
> "-pc_fieldsplit_schur_fact_type upper" uses
>
> K    -B^T
> 0     S
>

Yes.


> as preconditioner. Where K and S are inverted with the help of a KSP. So
> what I don't get is how to tweak the inner solvers correctly. For Example,
> In my own Implementation of preconditioned GMRES I also use the same
> structure as above as preconditioner and I take K^-1 to be approximated by
> the application of an AMG (in my case BoomerAMG) . Further I take S = B
> diag(K)^-1 B^T and again use an AMG to invert S. This results in my case in
> a good preconditioner.
>
> I tried now the same with this example and used
>
> -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type hypre
> -fieldsplit_1_ksp_type preonly -fieldsplit_1_pc_type hypre
>

1) Always start with 'lu' so that you can see what is responsible for
deterioration in convergence. I always start
    with the full Schur Complement factorization and accurate solves so
that it converges in 1 iterate. Then I
    back off some of it.

2) That preconditioner is definitely not optimal

3) For questions about convergence, always send

  -ksp_view -ksp_monitor_true_residual -ksp_converged_reason

and in this case turn on the monitors for the interior solves, e.g.

  -fieldsplit_1_ksp_monitor_true_residual

  Thanks,

     Matt


> However I get really slow convergence of the outer GMRES method.
>
> Can someone give me some insight into this?
>
> Best regards
>
> Elias
>
>
> On 28.05.2015 18:02, Matthew Knepley wrote:
>
> On Thu, May 28, 2015 at 10:47 AM, Elias Karabelas <
> elias.karabelas at medunigraz.at> wrote:
>
>> Dear Members,
>>
>> I want to solve a Block System arising from the Discretization of a
>> stabilized Finite Element Formulation of the Stokes System.
>>
>> I have the following Block Structure
>>
>> A     -B^T
>> B       C
>>
>> The Preconditioner I intend to use is a block preconditioner of the Form
>>
>> A    -B^T
>>         S
>>
>> where S is an approximation of the Schur Complement. For applying the
>> inverse of the schur complement I want to use a Stabilized Least Squares
>> Commutator in the form
>>
>> S^-1 = (B diag(Q)^-1 B^T + C_1)^-1 (B diag(Q)^-1 A diag(Q)^-1 B^T + C_2)
>> (B diag(Q)^-1 B^T + C_1)^-1
>>
>> where Q is the mass matrix and C_1 and C_2 are some additional
>> stabilization matrices.
>>
>> I got from the Manual, that I can use the PCFieldSplit preconditioner for
>> generating the general Block preconditioner as indicated above. And I also
>> found that I can define some arbitrary PC with PCSHELL. My question is, if
>> it is possible to use PCSHELL to define the action of S^-1 as indicated
>> above.
>>
>
> 1) Use FieldSplit is the right PC to start with. Make sure you can do
> something simple like
>
>   A -B^T
>       C + B diag(A)^{-1} B^T
>
> with it before we do the more complicated thing.
>
> 2) You will want to implement a PC for the (1,1) block. You can use a
> PCSHELL, which is simpler to setup, but
>     that means you will have to manually pull out the FieldSplit KSP and
> set it. If instead you define your own
>     PC implementation, its more boilerplate code, but you could specify
> this PC from the command line without
>     any FieldSplit specific code in your application.
>
> 3) Your PC will get two matrices, the MatSchurComplement, and the
> preconditioning matrix. If you set Q as the
>      preconditioning matrix, or really if you set
>
>     A   0
>     0   Q
>
> as the global preconditioning matrix, then the subsolve for (1,1) will get
> the Schur Complement and Q, and I think
> that is enough to build your Stabilized LSC PC.
>
> Let me know if this makes sense to you.
>
>   Thanks,
>
>     Matt
>
> Kind Regards
>> Elias Karabelas
>>
>> --
>> Elias Karabelas, Ph.D.
>>
>> Medical University of Graz
>> Institute of Biophysics
>> Harrachgasse 21/IV
>> 8010 Graz, Austria
>>
>> Phone: +43 316 380 7759
>> Email: elias.karabelas at medunigraz.at
>> Web  : http://forschung.medunigraz.at/fodok/staff?name=EliasKarabelas
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/4e8588b0/attachment.html>

From mfadams at lbl.gov  Fri May 29 09:02:51 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 29 May 2015 10:02:51 -0400
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
	<CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
	<7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>
Message-ID: <CADOhEh7QasskMc49woQiUiAbFBQRXVnca1OAFa7g9Gea9usMNw@mail.gmail.com>

Barry, It looks like you blew it.

On Wed, May 27, 2015 at 4:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Apparently all people with access to the PETSc repository (i.e.
> everyone in the world with internet access) have caught a rare disease that
> has caused their fingers to fall off, making them incapable of fixing this.
>
>    I have added all the PETSc enums to this file in master and next. Make
> sure you run make allfortranstubs to regenerate the stubs.
>
>    Barry
>
>
>
> > On May 27, 2015, at 2:15 PM, Mark Adams <mfadams at lbl.gov> wrote:
> >
> > FYI, this does not seem to be fixed.
> >
> > On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Looks like a bunch of these things are missing from
> lib/petsc/conf/bfort-petsc.txt  it should be listed under native
> >
> >   Barry
> >
> >
> >
> > > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > >
> > > Does bfort have to be told that PCCompositeType is an enum?
> > >
> > >    Matt "bfort's biggest fan"
> > >
> > > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > > It looks like fieldsplitf.c has an extra PetscToPointer.  type is
> already a pointer:
> > >
> > > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC
> pc,PCCompositeType *type, int *__ierr ){
> > > *__ierr = PCFieldSplitGetType(
> > >     (PC)PetscToPointer((pc) ),
> > >     (PCCompositeType* )PetscToPointer((type) ));
> > > }
> > >
> > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >    Likely need to run in the debugger to see what generates the error.
> Perhaps a missing ierr?
> > >
> > > Barry
> > >
> > >
> > > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > > >
> > > > I declare
> > > >
> > > >   PCCompositeType::pcctype
> > > >
> > > > But PCFieldSplitGetType does not seem to like it?
> > > >
> > > > Mark
> > > >
> > > > [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > > > [1]PETSC ERROR: Invalid pointer
> > > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
> > > > [1]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > > > [1]PETSC ERROR: Petsc Development GIT revision:
> v3.5.3-2702-g1f714da  GIT Date: 2015-04-16 11:32:06 -0500
> > > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named
> MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
> > > > [1]PETSC ERROR: Configure options --download-hypre=1
> --download-p4est --download-superlu=1 --download-mumps --download-scalapack
> --download-superlu_dist=1 --download-parmetis=1 --download-metis=1
> --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5
> --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
> > > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in
> /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/bb4802d1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 1694560 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/bb4802d1/attachment-0001.obj>

From hzhang at mcs.anl.gov  Fri May 29 09:12:45 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Fri, 29 May 2015 09:12:45 -0500
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CABzT7hM_VJcwWwkGkZ1A1-O0EM2-ShZjuWjE5b93nqo=_sWWGA@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CAGCphBuh9Nco8SvQhfsLC9AYnt9Uj_Nfyy_muaSxJRGhspdsUg@mail.gmail.com>
	<CABzT7hM_VJcwWwkGkZ1A1-O0EM2-ShZjuWjE5b93nqo=_sWWGA@mail.gmail.com>
Message-ID: <CAGCphBvkztzz-uG1=Yjja7Wr+=au7z-wwXiTm+AEsrj1s5bzJg@mail.gmail.com>

venkatesh:

> On Tue, May 26, 2015 at 9:02 PM, Hong <hzhang at mcs.anl.gov> wrote:
>
>> 'A serial job in MATLAB for the same matrices takes < 60GB. '
>> Can you run this case in serial? If so, try petsc, superlu or mumps to
>> make sure the matrix is non-singular.
>>
> B matrix is singular but I get my result in Petsc and Mumps for small
> matrices.
>

You are lucky to get something out of a singular matrix using LU
factorization, likely due to arithmetic roundoff. Is the obtained solution
useful?

Suggest reading petsc or slepc manual on how to dump out null space when a
matrix is singular.

Hong

>
>> Both mumps and superlu_dist show crash in MatFactorNumeric(). Mumps gives
>> error
>> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization
>> phase: Cannot allocate required memory 65710 megabytes.
>>
>> Does your code work for smaller problems?
>>
> Yes code works for small problems
>
>> Try using more processors?
>>
>> Why use such huge '-mat_mumps_icntl_14 200000' (percentage of estimated
>> workspace increase)? The default is 20. Try 40?
>>
>> Superlu_dist usually uses less memory than mumps, but it also crashes. I
>> guess something wrong with your matrix. Is it singular?
>>
> The B matrix is singular. So Super Lu cant be used is it ?
>
>
>> Run superlu_dist on a slightly smaller matrix with option
>> '-mat_superlu_dist_statprint' which displays memory usage info., e.g.,
>>
> Ok I will do that and check.
>
>> petsc/src/ksp/ksp/examples/tutorials (maint)
>> $ mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package
>> superlu_dist -mat_superlu_dist_statprint
>> Nonzeros in L       560
>> Nonzeros in U       560
>> nonzeros in L+U     1064
>> nonzeros in LSUB    248
>> NUMfact space (MB) sum(procs):  L\U 0.01 all 0.05
>> Total highmark (MB):  All 0.05 Avg 0.02 Max 0.02
>> EQUIL time             0.00
>> ROWPERM time           0.00
>> COLPERM time           0.00
>> SYMBFACT time          0.00
>> DISTRIBUTE time        0.00
>> FACTOR time            0.00
>> Factor flops 1.181000e+04 Mflops     4.80
>> SOLVE time             0.00
>> SOLVE time             0.00
>> Solve flops 2.194000e+03 Mflops     4.43
>> SOLVE time             0.00
>> Solve flops 2.194000e+03 Mflops     5.14
>> Norm of error 1.18018e-15 iterations 1
>>
>> Hong
>>
>>
>> On Tue, May 26, 2015 at 9:03 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>
>>> I am solving a large generalized Eigenvalue problem.
>>>
>>> I am getting the following error which is attached, after giving the
>>> command:
>>>
>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64 -hosts
>>> compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2 b72t
>>> -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>> 200000
>>>
>>> IT IS impossible to allocate so much memory per processor.. it is asking
>>> like around 70 GB per processor.
>>>
>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>
>>> After trying out superLU_dist, I have attached the error there also
>>> (segmentation error).
>>>
>>> Kindly help me.
>>>
>>> Venkatesh
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/cc563a46/attachment.html>

From venkateshgk.j at gmail.com  Fri May 29 10:21:56 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 29 May 2015 20:51:56 +0530
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CAGCphBvkztzz-uG1=Yjja7Wr+=au7z-wwXiTm+AEsrj1s5bzJg@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CAGCphBuh9Nco8SvQhfsLC9AYnt9Uj_Nfyy_muaSxJRGhspdsUg@mail.gmail.com>
	<CABzT7hM_VJcwWwkGkZ1A1-O0EM2-ShZjuWjE5b93nqo=_sWWGA@mail.gmail.com>
	<CAGCphBvkztzz-uG1=Yjja7Wr+=au7z-wwXiTm+AEsrj1s5bzJg@mail.gmail.com>
Message-ID: <CABzT7hNJ+cSccq75-g-_FrXDnh1A_3fWBxy2wAirCnkQSXyvWQ@mail.gmail.com>

Thanks. So I what other direct solver I can use for this singular B ? QR is
a good option but what to do ?

I will read the manual on dumping out null space.

Venkatesh

On Fri, May 29, 2015 at 7:42 PM, Hong <hzhang at mcs.anl.gov> wrote:

> venkatesh:
>
>> On Tue, May 26, 2015 at 9:02 PM, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> 'A serial job in MATLAB for the same matrices takes < 60GB. '
>>> Can you run this case in serial? If so, try petsc, superlu or mumps to
>>> make sure the matrix is non-singular.
>>>
>> B matrix is singular but I get my result in Petsc and Mumps for small
>> matrices.
>>
>
> You are lucky to get something out of a singular matrix using LU
> factorization, likely due to arithmetic roundoff. Is the obtained solution
> useful?
>
> Suggest reading petsc or slepc manual on how to dump out null space when a
> matrix is singular.
>
> Hong
>
>>
>>> Both mumps and superlu_dist show crash in MatFactorNumeric(). Mumps
>>> gives error
>>> [16]PETSC ERROR: Error reported by MUMPS in numerical factorization
>>> phase: Cannot allocate required memory 65710 megabytes.
>>>
>>> Does your code work for smaller problems?
>>>
>> Yes code works for small problems
>>
>>> Try using more processors?
>>>
>>> Why use such huge '-mat_mumps_icntl_14 200000' (percentage of estimated
>>> workspace increase)? The default is 20. Try 40?
>>>
>>> Superlu_dist usually uses less memory than mumps, but it also crashes. I
>>> guess something wrong with your matrix. Is it singular?
>>>
>> The B matrix is singular. So Super Lu cant be used is it ?
>>
>>
>>> Run superlu_dist on a slightly smaller matrix with option
>>> '-mat_superlu_dist_statprint' which displays memory usage info., e.g.,
>>>
>> Ok I will do that and check.
>>
>>> petsc/src/ksp/ksp/examples/tutorials (maint)
>>> $ mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package
>>> superlu_dist -mat_superlu_dist_statprint
>>> Nonzeros in L       560
>>> Nonzeros in U       560
>>> nonzeros in L+U     1064
>>> nonzeros in LSUB    248
>>> NUMfact space (MB) sum(procs):  L\U 0.01 all 0.05
>>> Total highmark (MB):  All 0.05 Avg 0.02 Max 0.02
>>> EQUIL time             0.00
>>> ROWPERM time           0.00
>>> COLPERM time           0.00
>>> SYMBFACT time          0.00
>>> DISTRIBUTE time        0.00
>>> FACTOR time            0.00
>>> Factor flops 1.181000e+04 Mflops     4.80
>>> SOLVE time             0.00
>>> SOLVE time             0.00
>>> Solve flops 2.194000e+03 Mflops     4.43
>>> SOLVE time             0.00
>>> Solve flops 2.194000e+03 Mflops     5.14
>>> Norm of error 1.18018e-15 iterations 1
>>>
>>> Hong
>>>
>>>
>>> On Tue, May 26, 2015 at 9:03 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>
>>>> I am solving a large generalized Eigenvalue problem.
>>>>
>>>> I am getting the following error which is attached, after giving the
>>>> command:
>>>>
>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>> 200000
>>>>
>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>> asking like around 70 GB per processor.
>>>>
>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>
>>>> After trying out superLU_dist, I have attached the error there also
>>>> (segmentation error).
>>>>
>>>> Kindly help me.
>>>>
>>>> Venkatesh
>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/5ffd1109/attachment.html>

From balay at mcs.anl.gov  Fri May 29 10:23:45 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 29 May 2015 10:23:45 -0500
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <CADOhEh7QasskMc49woQiUiAbFBQRXVnca1OAFa7g9Gea9usMNw@mail.gmail.com>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
	<CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
	<7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>
	<CADOhEh7QasskMc49woQiUiAbFBQRXVnca1OAFa7g9Gea9usMNw@mail.gmail.com>
Message-ID: <alpine.LFD.2.11.1505291020410.7733@asterix>

I tried building on edison - and can't reproduce this problem.

./configure --COPTFLAGS="-fast -no-ipo" --CXXOPTFLAGS="-fast -no-ipo" --FOPTFLAGS="-fast -no-ipo"  --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn  --with-fortranlib-autodetect=0  --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun  LIBS=-lstdc++ --with-64-bit-indices PETSC_DIR=$PWD PETSC_ARCH=arch-xc30-opt64-intel --prefix=$HOME/soft/test

The error message you get is puzzling..

> Unable to find bfort-base.txt config file in /global/homes/m/madams/petsc_install/petsc/arch-xc30-opt64-intel/share*******

Can you attempt a build from a 'clean' repo?

rm -rf arch*
git clean -f -d -x
./configure....

Satish


On Fri, 29 May 2015, Mark Adams wrote:

> Barry, It looks like you blew it.
> 
> On Wed, May 27, 2015 at 4:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> >
> >    Apparently all people with access to the PETSc repository (i.e.
> > everyone in the world with internet access) have caught a rare disease that
> > has caused their fingers to fall off, making them incapable of fixing this.
> >
> >    I have added all the PETSc enums to this file in master and next. Make
> > sure you run make allfortranstubs to regenerate the stubs.
> >
> >    Barry
> >
> >
> >
> > > On May 27, 2015, at 2:15 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > >
> > > FYI, this does not seem to be fixed.
> > >
> > > On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > >   Looks like a bunch of these things are missing from
> > lib/petsc/conf/bfort-petsc.txt  it should be listed under native
> > >
> > >   Barry
> > >
> > >
> > >
> > > > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com>
> > wrote:
> > > >
> > > > Does bfort have to be told that PCCompositeType is an enum?
> > > >
> > > >    Matt "bfort's biggest fan"
> > > >
> > > > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > > > It looks like fieldsplitf.c has an extra PetscToPointer.  type is
> > already a pointer:
> > > >
> > > > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC
> > pc,PCCompositeType *type, int *__ierr ){
> > > > *__ierr = PCFieldSplitGetType(
> > > >     (PC)PetscToPointer((pc) ),
> > > >     (PCCompositeType* )PetscToPointer((type) ));
> > > > }
> > > >
> > > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov>
> > wrote:
> > > >
> > > >    Likely need to run in the debugger to see what generates the error.
> > Perhaps a missing ierr?
> > > >
> > > > Barry
> > > >
> > > >
> > > > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > > > >
> > > > > I declare
> > > > >
> > > > >   PCCompositeType::pcctype
> > > > >
> > > > > But PCFieldSplitGetType does not seem to like it?
> > > > >
> > > > > Mark
> > > > >
> > > > > [1]PETSC ERROR: --------------------- Error Message
> > --------------------------------------------------------------
> > > > > [1]PETSC ERROR: Invalid pointer
> > > > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
> > > > > [1]PETSC ERROR: See
> > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > > > > [1]PETSC ERROR: Petsc Development GIT revision:
> > v3.5.3-2702-g1f714da  GIT Date: 2015-04-16 11:32:06 -0500
> > > > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named
> > MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
> > > > > [1]PETSC ERROR: Configure options --download-hypre=1
> > --download-p4est --download-superlu=1 --download-mumps --download-scalapack
> > --download-superlu_dist=1 --download-parmetis=1 --download-metis=1
> > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5
> > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
> > > > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in
> > /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which their
> > experiments lead.
> > > > -- Norbert Wiener
> > >
> > >
> >
> >
> 


From mfadams at lbl.gov  Fri May 29 12:05:47 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 29 May 2015 13:05:47 -0400
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <alpine.LFD.2.11.1505291020410.7733@asterix>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
	<CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
	<7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>
	<CADOhEh7QasskMc49woQiUiAbFBQRXVnca1OAFa7g9Gea9usMNw@mail.gmail.com>
	<alpine.LFD.2.11.1505291020410.7733@asterix>
Message-ID: <CADOhEh6UNM7Yx2C6BEJ2w91e5QtZkcH7j=oX6=83PJC1aiNcTg@mail.gmail.com>

Humm, I cleaned everything, and even cut and pasted your configure line,
same thing.  I am now trying with a fresh petsc clone.

I add these modules:

  module load adios/1.6.0
  module load cray-hdf5-parallel
  module load cmake/2.8.11.2
  module unload darshan

and I do not see this bfort-base.txt file:

10:03 edison03 master ~/petsc_install/petsc$ ll arch-xc30-opt64-intel/share
total 32
-rw-r--r-- 1 madams madams 4755 May 29 09:52 basedefs.txt
-rw-r--r-- 1 madams madams  326 May 29 09:52 blueball.gif
drwxr-x--- 2 madams madams  512 May 29 09:52 doctext
-rw-r--r-- 1 madams madams  326 May 29 09:52 greenball.gif
-rw-r--r-- 1 madams madams 1625 May 29 09:52 html.def
-rw-r--r-- 1 madams madams 1668 May 29 09:52 latex.def
drwxr-x--- 3 madams madams  512 May 29 09:52 man
-rw-r--r-- 1 madams madams   79 May 29 09:52 next.gif
-rw-r--r-- 1 madams madams  287 May 29 09:52 next.xbm
-rw-r--r-- 1 madams madams 1482 May 29 09:52 nroff.def
-rw-r--r-- 1 madams madams   79 May 29 09:52 previous.gif
-rw-r--r-- 1 madams madams  299 May 29 09:52 previous.xbm
-rw-r--r-- 1 madams madams 7353 May 29 09:52 pstoppm.ps
-rw-r--r-- 1 madams madams  326 May 29 09:52 purpleball.gif
-rw-r--r-- 1 madams madams  326 May 29 09:52 redball.gif
-rw-r--r-- 1 madams madams  283 May 29 09:52 refman.def
-rw-r--r-- 1 madams madams 3151 May 29 09:52 refman.sty
-rw-r--r-- 1 madams madams   74 May 29 09:52 up.gif
-rw-r--r-- 1 madams madams  281 May 29 09:52 up.xbm
-rw-r--r-- 1 madams madams  324 May 29 09:52 yellowball.gif




On Fri, May 29, 2015 at 11:23 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> I tried building on edison - and can't reproduce this problem.
>
> ./configure --COPTFLAGS="-fast -no-ipo" --CXXOPTFLAGS="-fast -no-ipo"
> --FOPTFLAGS="-fast -no-ipo"  --with-ssl=0 --with-cc=cc
> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0
> --with-debugging=0 --with-fc=ftn  --with-fortranlib-autodetect=0
> --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun  LIBS=-lstdc++
> --with-64-bit-indices PETSC_DIR=$PWD PETSC_ARCH=arch-xc30-opt64-intel
> --prefix=$HOME/soft/test
>
> The error message you get is puzzling..
>
> > Unable to find bfort-base.txt config file in
> /global/homes/m/madams/petsc_install/petsc/arch-xc30-opt64-intel/share*******
>
> Can you attempt a build from a 'clean' repo?
>
> rm -rf arch*
> git clean -f -d -x
> ./configure....
>
> Satish
>
>
> On Fri, 29 May 2015, Mark Adams wrote:
>
> > Barry, It looks like you blew it.
> >
> > On Wed, May 27, 2015 at 4:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > >
> > >    Apparently all people with access to the PETSc repository (i.e.
> > > everyone in the world with internet access) have caught a rare disease
> that
> > > has caused their fingers to fall off, making them incapable of fixing
> this.
> > >
> > >    I have added all the PETSc enums to this file in master and next.
> Make
> > > sure you run make allfortranstubs to regenerate the stubs.
> > >
> > >    Barry
> > >
> > >
> > >
> > > > On May 27, 2015, at 2:15 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > > >
> > > > FYI, this does not seem to be fixed.
> > > >
> > > > On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > > >
> > > >   Looks like a bunch of these things are missing from
> > > lib/petsc/conf/bfort-petsc.txt  it should be listed under native
> > > >
> > > >   Barry
> > > >
> > > >
> > > >
> > > > > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com>
> > > wrote:
> > > > >
> > > > > Does bfort have to be told that PCCompositeType is an enum?
> > > > >
> > > > >    Matt "bfort's biggest fan"
> > > > >
> > > > > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov>
> wrote:
> > > > > It looks like fieldsplitf.c has an extra PetscToPointer.  type is
> > > already a pointer:
> > > > >
> > > > > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC
> > > pc,PCCompositeType *type, int *__ierr ){
> > > > > *__ierr = PCFieldSplitGetType(
> > > > >     (PC)PetscToPointer((pc) ),
> > > > >     (PCCompositeType* )PetscToPointer((type) ));
> > > > > }
> > > > >
> > > > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov>
> > > wrote:
> > > > >
> > > > >    Likely need to run in the debugger to see what generates the
> error.
> > > Perhaps a missing ierr?
> > > > >
> > > > > Barry
> > > > >
> > > > >
> > > > > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > > > > >
> > > > > > I declare
> > > > > >
> > > > > >   PCCompositeType::pcctype
> > > > > >
> > > > > > But PCFieldSplitGetType does not seem to like it?
> > > > > >
> > > > > > Mark
> > > > > >
> > > > > > [1]PETSC ERROR: --------------------- Error Message
> > > --------------------------------------------------------------
> > > > > > [1]PETSC ERROR: Invalid pointer
> > > > > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
> > > > > > [1]PETSC ERROR: See
> > > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> shooting.
> > > > > > [1]PETSC ERROR: Petsc Development GIT revision:
> > > v3.5.3-2702-g1f714da  GIT Date: 2015-04-16 11:32:06 -0500
> > > > > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named
> > > MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
> > > > > > [1]PETSC ERROR: Configure options --download-hypre=1
> > > --download-p4est --download-superlu=1 --download-mumps
> --download-scalapack
> > > --download-superlu_dist=1 --download-parmetis=1 --download-metis=1
> > > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5
> > > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
> > > > > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in
> > > /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> their
> > > experiments lead.
> > > > > -- Norbert Wiener
> > > >
> > > >
> > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/efa61bae/attachment.html>

From venkateshgk.j at gmail.com  Fri May 29 12:16:25 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Fri, 29 May 2015 22:46:25 +0530
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CADZ9mvkihmmyMvjVDW5GV7FsFj0XPwy=o6maV+SbM4KuKZKEvQ@mail.gmail.com>
	<CABzT7hNfgcO_hknFxmVpuZx5GZUbp0mEmR6Dmmjqp7myWHOqBw@mail.gmail.com>
	<CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
Message-ID: <CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>

Hi Matt, users,

Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get
into the segmentation error if I increase my matrix size.

Can you suggest other software for direct solver for QR in parallel since
as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am
attaching the working version mumps log.

My matrix size here is around 47000x47000. If I am not wrong, the memory
usage per core is 272MB.

Can you tell me if I am wrong ? or really if its light on memory for this
matrix ?

Thanks
cheers,
Venkatesh

On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <matt.landreman at gmail.com>
wrote:

> Dear Venkatesh,
>
> As you can see in the error log, you are now getting a segmentation fault,
> which is almost certainly a separate issue from the info(1)=-9 memory
> problem you had previously. Here is one idea which may or may not help.
> I've used mumps on the NERSC Edison system, and I found that I sometimes
> get segmentation faults when using the default Intel compiler. When I
> switched to the cray compiler the problem disappeared. So you could perhaps
> try a different compiler if one is available on your system.
>
> Matt
> On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j at gmail.com> wrote:
>
>> Hi Matt,
>>
>> I did what you told and read the manual of that CNTL parameters. I solve
>> for that with CNTL(1)=1e-4. It is working.
>>
>> But it was a test matrix with size 46000x46000. Actual matrix size is
>> 108900x108900 and will increase in the future.
>>
>> I get this error of memory allocation failed. And the binary matrix size
>> of A is 20GB and B is 5 GB.
>>
>> Now I submit this in 240 processors each 4 GB RAM and also in 128
>> Processors with total 512 GB RAM.
>>
>> In both the cases, it fails with the following error like memory is not
>> enough. But for 90000x90000 size it had run serially in Matlab with <256 GB
>> RAM.
>>
>> Kindly let me know.
>>
>> Venkatesh
>>
>> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <matt.landreman at gmail.com
>> > wrote:
>>
>>> Hi Venkatesh,
>>>
>>> I've struggled a bit with mumps memory allocation too.  I think the
>>> behavior of mumps is roughly the following. First, in the "analysis step",
>>> mumps computes a minimum memory required based on the structure of nonzeros
>>> in the matrix.  Then when it actually goes to factorize the matrix, if it
>>> ever encounters an element smaller than CNTL(1) (default=0.01) in the
>>> diagonal of a sub-matrix it is trying to factorize, it modifies the
>>> ordering to avoid the small pivot, which increases the fill-in (hence
>>> memory needed).  ICNTL(14) sets the margin allowed for this unanticipated
>>> fill-in.  Setting ICNTL(14)=200000 as in your email is not the solution,
>>> since this means mumps asks for a huge amount of memory at the start.
>>> Better would be to lower CNTL(1) or (I think) use static pivoting
>>> (CNTL(4)).  Read the section in the mumps manual about these CNTL
>>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the
>>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14).
>>>
>>> Also, I recommend running with ICNTL(4)=3 to display diagnostics. Look
>>> for the line in standard output that says "TOTAL     space in MBYTES for IC
>>> factorization".  This is the amount of memory that mumps is trying to
>>> allocate, and for the default ICNTL(14), it should be similar to matlab's
>>> need.
>>>
>>> Hope this helps,
>>> -Matt Landreman
>>> University of Maryland
>>>
>>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <venkateshgk.j at gmail.com>
>>> wrote:
>>>
>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>
>>>> I am solving a large generalized Eigenvalue problem.
>>>>
>>>> I am getting the following error which is attached, after giving the
>>>> command:
>>>>
>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>> 200000
>>>>
>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>> asking like around 70 GB per processor.
>>>>
>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>
>>>> After trying out superLU_dist, I have attached the error there also
>>>> (segmentation error).
>>>>
>>>> Kindly help me.
>>>>
>>>> Venkatesh
>>>>
>>>>
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/49452c18/attachment-0001.html>
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
Entering ZMUMPS driver with JOB, N, NZ =   1       46656              0

 ZMUMPS 4.10.0        
L U Solver for unsymmetric matrices
Type of parallelism: Working host

 ****** ANALYSIS STEP ********

 ** Max-trans not allowed because matrix is distributed
 ... Structural symmetry (in percent)=   75
 Density: NBdense, Average, Median   =    0 6488 5186
 Ordering based on METIS 
 A root of estimated size        17063  has been selected for Scalapack.

Leaving analysis phase with  ...
INFOG(1)                                       =               0
INFOG(2)                                       =               0
 -- (20) Number of entries in factors (estim.) =       987031396
 --  (3) Storage of factors  (REAL, estimated) =      1059159558
 --  (4) Storage of factors  (INT , estimated) =         9996429
 --  (5) Maximum frontal size      (estimated) =           17063
 --  (6) Number of nodes in the tree           =             226
 -- (32) Type of analysis effectively used     =               1
 --  (7) Ordering option effectively used      =               5
ICNTL(6) Maximum transversal option            =               0
ICNTL(7) Pivot order option                    =               7
Percentage of memory relaxation (effective)    =              35
Number of level 2 nodes                        =             209
Number of split nodes                          =              12
RINFOG(1) Operations during elimination (estim)=   1.182D+13
Distributed matrix entry format (ICNTL(18))    =               3
 ** Rank of proc needing largest memory in IC facto        :         0
 ** Estimated corresponding MBYTES for IC facto            :      6275
 ** Estimated avg. MBYTES per work. proc at facto (IC)     :      1478
 ** TOTAL     space in MBYTES for IC factorization         :    177411
 ** Rank of proc needing largest memory for OOC facto      :         0
 ** Estimated corresponding MBYTES for OOC facto           :      6345
 ** Estimated avg. MBYTES per work. proc at facto (OOC)    :      1484
 ** TOTAL     space in MBYTES for OOC factorization        :    178120
Entering ZMUMPS driver with JOB, N, NZ =   2       46656      241356672

 ****** FACTORIZATION STEP ********


 GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...
 NUMBER OF WORKING PROCESSES              =         120
 OUT-OF-CORE OPTION (ICNTL(22))           =           0
 REAL SPACE FOR FACTORS                   =  1059159558
 INTEGER SPACE FOR FACTORS                =     9996429
 MAXIMUM FRONTAL SIZE (ESTIMATED)         =       17063
 NUMBER OF NODES IN THE TREE              =         226
 Convergence error after scaling for ONE-NORM (option 7/8)   = 0.93D+00
 Maximum effective relaxed size of S              =   305345800
 Average effective relaxed size of S              =    29225972

 REDISTRIB: TOTAL DATA LOCAL/SENT         =    71295798   -58336695
 GLOBAL TIME FOR MATRIX DISTRIBUTION       =     41.2996
 ** Memory relaxation parameter ( ICNTL(14)  )            :        35
 ** Rank of processor needing largest memory in facto     :         0
 ** Space in MBYTES used by this processor for facto      :      6275
 ** Avg. Space in MBYTES per working proc during facto    :      1478

 ELAPSED TIME FOR FACTORIZATION           =    138.3655
 Maximum effective space used in S   (KEEP8(67)   =   118089792
 Average effective space used in S   (KEEP8(67)   =    17422614
 ** EFF Min: Rank of processor needing largest memory :         0
 ** EFF Min: Space in MBYTES used by this processor   :      3310
 ** EFF Min: Avg. Space in MBYTES per working proc    :      1289

 GLOBAL STATISTICS 
 RINFOG(2)  OPERATIONS IN NODE ASSEMBLY   = 3.060D+10
 ------(3)  OPERATIONS IN NODE ELIMINATION= 1.164D+13
 INFOG (9)  REAL SPACE FOR FACTORS        =   980699854
 INFOG(10)  INTEGER SPACE FOR FACTORS     =     9059190
 INFOG(11)  MAXIMUM FRONT SIZE            =       17063
 INFOG(29)  NUMBER OF ENTRIES IN FACTORS  =   689553885
 INFOG(12) NB OF OFF DIAGONAL PIVOTS      =           4
 INFOG(13)  NUMBER OF DELAYED PIVOTS      =           1
 INFOG(14)  NUMBER OF MEMORY COMPRESS     =         782
 KEEP8(108) Extra copies IP stacking      =           0
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
Entering ZMUMPS driver with JOB, N, NZ =   3       46656      241356672


 ****** SOLVE & CHECK STEP ********


 STATISTICS PRIOR SOLVE PHASE     ...........
 NUMBER OF RIGHT-HAND-SIDES                    =           1
 BLOCKING FACTOR FOR MULTIPLE RHS              =           1
 ICNTL (9)                                     =           1
  --- (10)                                     =           0
  --- (11)                                     =           0
  --- (20)                                     =           0
  --- (21)                                     =           1
  --- (30)                                     =           0
 ** Rank of processor needing largest memory in solve     :         0
 ** Space in MBYTES used by this processor for solve      :      5105
 ** Avg. Space in MBYTES per working proc during solve    :       626
 Number of iterations of the method: 10
 Number of linear iterations of the method: 99
 Solution method: krylovschur

 Number of requested eigenvalues: 3
 Stopping condition: tol=1e-08, maxit=5184
 Number of converged approximate eigenpairs: 9

           k          ||Ax-kBx||/||kx||
   ----------------- ------------------
 -0.010645+0.000000 i      1.78065
 -0.010672-0.000000 i     0.358554
 -0.011290+0.000000 i     0.360215
 -0.011517+0.000000 i    0.0574819
 -0.012048-0.000000 i    0.0545338
 -0.014239-0.000000 i   0.00326724
 -0.018303-0.000000 i    0.0152795
 -0.024691-0.000000 i   0.00253268
 -0.033922-0.000000 i    0.0041931

Entering ZMUMPS driver with JOB, N, NZ =  -2       46656      241356672
************************************************************************************************************************
***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***
************************************************************************************************************************

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

./ex7 on a linux-intel named nid01404 with 120 processors, by esdveng Thu May 28 06:00:20 2015
Using Petsc Release Version 3.5.3, Jan, 31, 2015 

                         Max       Max/Min        Avg      Total 
Time (sec):           2.872e+03      1.00001   2.872e+03
Objects:              1.030e+02      1.01980   1.017e+02
Flops:                1.921e+09     80.08812   5.266e+08  6.320e+10
Flops/sec:            6.691e+05     80.08753   1.834e+05  2.201e+07
Memory:               2.726e+08     79.01630              1.576e+10
MPI Messages:         5.523e+03     21.12046   2.531e+03  3.038e+05
MPI Message Lengths:  2.432e+09   1394.39160   1.845e+04  5.603e+09
MPI Reductions:       3.023e+03      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.8715e+03 100.0%  6.3197e+10 100.0%  3.038e+05 100.0%  1.845e+04      100.0%  3.022e+03 100.0% 

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flops: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flops in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

ThreadCommRunKer       1 1.0 1.0540e-03 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
ThreadCommBarrie       1 1.0 4.0531e-06 4.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMult              117 1.0 2.7497e+00124.8 1.90e+09937.7 1.1e+05 5.8e+03 0.0e+00  0 96 37 12  0   0 96 37 12  0 22023
MatSolve              99 1.0 8.6172e+01 1.0 0.00e+00 0.0 1.7e+05 6.3e+02 1.0e+02  3  0 57  2  3   3  0 57  2  3     0
MatLUFactorSym         1 1.0 2.5466e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 89  0  0  0  0  89  0  0  0  0     0
MatLUFactorNum         1 1.0 1.8082e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  6  0  0  0  0   6  0  0  0  0     0
MatConvert             1 1.0 1.2173e-01 5.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin       3 1.0 3.8287e+0110.8 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatAssemblyEnd         3 1.0 4.6185e+00 1.0 0.00e+00 0.0 1.6e+04 7.1e+02 4.2e+01  0  0  5  0  1   0  0  5  0  1     0
MatGetRow            778 1.0 5.5872e-0215.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 1.0 1.4498e-03160.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 3.1960e-03 7.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLoad                2 1.0 4.7483e+01 1.0 0.00e+00 0.0 9.0e+03 5.4e+05 4.6e+01  2  0  3 86  2   2  0  3 86  2     0
MatAXPY                1 1.0 5.2885e+00 1.0 0.00e+00 0.0 7.9e+03 7.1e+02 2.2e+01  0  0  3  0  1   0  0  3  0  1     0
VecNorm               18 1.0 8.2703e-03 2.0 5.60e+04 1.0 0.0e+00 0.0e+00 1.8e+01  0  0  0  0  1   0  0  0  0  1   812
VecCopy               20 1.0 1.8883e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               113 1.0 7.5676e-03 9.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                9 1.0 6.5203e-03 1.3 2.80e+04 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   515
VecScatterBegin      315 1.0 9.9498e-02 7.1 0.00e+00 0.0 2.8e+05 2.7e+03 9.9e+01  0  0 93 14  3   0  0 93 14  3     0
VecScatterEnd        216 1.0 2.0806e-0141.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
EPSSetUp               1 1.0 2.7329e+03 1.0 0.00e+00 0.0 7.9e+03 7.1e+02 6.7e+01 95  0  3  0  2  95  0  3  0  2     0
EPSSolve               1 1.0 2.8219e+03 1.0 1.62e+0974.8 2.6e+05 2.1e+03 2.8e+03 98 72 85 10 93  98 72 85 10 93    16
STSetUp                1 1.0 2.7329e+03 1.0 0.00e+00 0.0 7.9e+03 7.1e+02 4.6e+01 95  0  3  0  2  95  0  3  0  2     0
STApply               99 1.0 8.8469e+01 1.0 1.60e+09 0.0 2.5e+05 2.2e+03 3.0e+02  3 68 82 10 10   3 68 82 10 10   488
STMatSolve            99 1.0 8.8451e+01 1.0 0.00e+00 0.0 1.7e+05 6.3e+02 3.0e+02  3  0 57  2 10   3  0 57  2 10     0
BVCopy                19 1.0 1.1969e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 5.6e+01  0  0  0  0  2   0  0  0  0  2     0
BVMult               205 1.0 4.4088e-02 1.1 1.32e+07 1.0 0.0e+00 0.0e+00 3.9e+02  0  3  0  0 13   0  3  0  0 13 35896
BVDot                194 1.0 7.7536e-02 1.2 8.59e+06 1.0 0.0e+00 0.0e+00 1.9e+02  0  2  0  0  6   0  2  0  0  6 13281
BVOrthogonalize      100 1.0 1.9655e-01 1.0 1.66e+07 1.0 0.0e+00 0.0e+00 1.6e+03  0  3  0  0 51   0  3  0  0 51 10110
BVScale              100 1.0 6.5482e-03 1.1 1.56e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2850
BVSetRandom            1 1.0 7.0076e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSSolve               10 1.0 3.2166e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSVectors             20 1.0 3.5219e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSOther               10 1.0 1.0614e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetUp               1 1.0 9.0599e-06 4.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve              99 1.0 8.8438e+01 1.0 0.00e+00 0.0 1.7e+05 6.3e+02 3.0e+02  3  0 57  2 10   3  0 57  2 10     0
PCSetUp                1 1.0 2.7275e+03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.6e+01 95  0  0  0  1  95  0  0  0  1     0
PCApply               99 1.0 8.6174e+01 1.0 0.00e+00 0.0 1.7e+05 6.3e+02 1.0e+02  3  0 57  2  3   3  0 57  2  3     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     3              2         1504     0
              Matrix    17             17      3542976     0
              Vector    54             54      1459280     0
      Vector Scatter     6              6         7200     0
           Index Set    13             13       122048     0
Eigenvalue Problem Solver     1              1         2156     0
         PetscRandom     1              1          648     0
  Spectral Transform     1              1          840     0
       Basis Vectors     1              1        10744     0
              Region     1              1          648     0
       Direct solver     1              1        25272     0
       Krylov Solver     1              1         1160     0
      Preconditioner     1              1         1096     0
========================================================================================================================
Average time to get PetscTime(): 1.19209e-07
Average time for MPI_Barrier(): 2.94209e-05
Average time for zero size MPI_Send(): 1.11659e-06
#PETSc Option Table entries:
-eps_nev 3
-eps_target 0.5
-f1 a72t
-f2 b72t
-log_summary
-mat_mumps_cntl_1 1e-6
-mat_mumps_icntl_4 2
-st_ksp_type preonly
-st_pc_factor_mat_solver_package mumps
-st_pc_type lu
-st_type sinvert
#End of PETSc Option Table entries
Compiled with FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 16 sizeof(PetscInt) 4
Configure options: --known-level1-dcache-size=32768 --known-level1-dcache-linesize=32 --known-level1-dcache-assoc=0 --known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 PETSC_ARCH=linux-intel -with-blas-lapack-dir=/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64/ --with-mpi-dir=/opt/cray/mpt/7.0.5/gni/mpich2-intel/140 --with-scalar-type=complex --with-fortran-kernels=1 -known-mpi-shared-libraries=0 --with-large-file-io=1 --with-64-bit-indices=0 --with-batch FC=ifort --with-valgrind-dir=/home/proj/14/esdveng/apps/valgrind-3.10.1 --download-mumps=/home/proj/14/esdveng/apps/MUMPS_4.10.0-p3.tar.gz --download-scalapack=/home/proj/14/esdveng/apps/scalapack-2.0.2.tgz --download-blacs=/home/proj/14/esdveng/apps/blacs-dev.tar.gz --download-parmetis=/home/proj/14/esdveng/apps/parmetis-4.0.2-p5.tar.gz --download-metis=/home/proj/14/esdveng/apps/metis-5.0.2-p3.tar.gz --download-cmake=/home/proj/14/esdveng/apps/cmake-2.8.12.2.tar.gz
-----------------------------------------
Libraries compiled on Tue May 19 06:39:06 2015 on login4 
Machine characteristics: Linux-3.0.101-0.31.1_1.0502.8394-cray_ari_s-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /mnt/lustre/esd2/esdveng/petsc-3.5.3
Using PETSc arch: linux-intel
-----------------------------------------

Using C compiler: gcc  -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: ifort  -fPIC -g   ${FOPTFLAGS} ${FFLAGS} 
-----------------------------------------

Using include paths: -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/include -I/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/include -I/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/include
-----------------------------------------

Using C linker: gcc
Using Fortran linker: ifort
Using libraries: -Wl,-rpath,/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -L/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -lpetsc -Wl,-rpath,/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -L/mnt/lustre/esd2/esdveng/petsc-3.5.3/linux-intel/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -lparmetis -lmetis -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/lib -L/opt/cray/mpt/7.0.5/gni/mpich2-intel/140/lib -lmpich -lssl -luuid -lpthread -lrt -ldl -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lpthread -lirc_s -lm -lstdc++ -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -lstdc++ -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -L/opt/intel/composer_xe_2015.1.133/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -L/opt/intel/composer_xe_2015.1.133/tbb/lib/intel64/gcc4.1 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -ldl -lgcc_s -ldl  
-----------------------------------------

Application 407926 resources: utime ~342719s, stime ~1191s, Rss ~7065700, inblocks ~12131942, outblocks ~35052

From mfadams at lbl.gov  Fri May 29 12:20:32 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 29 May 2015 13:20:32 -0400
Subject: [petsc-users] error using GetType in FORTRAN
In-Reply-To: <CADOhEh6UNM7Yx2C6BEJ2w91e5QtZkcH7j=oX6=83PJC1aiNcTg@mail.gmail.com>
References: <CADOhEh41nNOpLb8HUX=aENbTQBncJdtzxkWGowPCZqiwQ+Jr_w@mail.gmail.com>
	<79ABF344-22F0-4FE8-9DF8-4793D8D0265F@mcs.anl.gov>
	<CADOhEh4JP+SwXzufSKOfoo4QsPZE1xuaRKYkY1fKHY0T-wCtAQ@mail.gmail.com>
	<CAMYG4GmOC8Y=MKVVYgbDsYA_x4wpoqXs0M=r85pJbse6oHCstA@mail.gmail.com>
	<9F934B14-8082-49E5-AB36-53D4F07E5217@mcs.anl.gov>
	<CADOhEh7OwgpvJ=nFZwCUcQhoNaAo2DuE22z6SePzEU2+CZkD9A@mail.gmail.com>
	<7CE354EF-8366-4627-AD40-FBA09F3BC375@mcs.anl.gov>
	<CADOhEh7QasskMc49woQiUiAbFBQRXVnca1OAFa7g9Gea9usMNw@mail.gmail.com>
	<alpine.LFD.2.11.1505291020410.7733@asterix>
	<CADOhEh6UNM7Yx2C6BEJ2w91e5QtZkcH7j=oX6=83PJC1aiNcTg@mail.gmail.com>
Message-ID: <CADOhEh6Do_9jufY4zEhFNx7khEJbxNNDyPw+jWkA6ChfzFADWQ@mail.gmail.com>

Well I used a fresh petsc clone and that worked.

On Fri, May 29, 2015 at 1:05 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Humm, I cleaned everything, and even cut and pasted your configure line,
> same thing.  I am now trying with a fresh petsc clone.
>
> I add these modules:
>
>   module load adios/1.6.0
>   module load cray-hdf5-parallel
>   module load cmake/2.8.11.2
>   module unload darshan
>
> and I do not see this bfort-base.txt file:
>
> 10:03 edison03 master ~/petsc_install/petsc$ ll arch-xc30-opt64-intel/share
> total 32
> -rw-r--r-- 1 madams madams 4755 May 29 09:52 basedefs.txt
> -rw-r--r-- 1 madams madams  326 May 29 09:52 blueball.gif
> drwxr-x--- 2 madams madams  512 May 29 09:52 doctext
> -rw-r--r-- 1 madams madams  326 May 29 09:52 greenball.gif
> -rw-r--r-- 1 madams madams 1625 May 29 09:52 html.def
> -rw-r--r-- 1 madams madams 1668 May 29 09:52 latex.def
> drwxr-x--- 3 madams madams  512 May 29 09:52 man
> -rw-r--r-- 1 madams madams   79 May 29 09:52 next.gif
> -rw-r--r-- 1 madams madams  287 May 29 09:52 next.xbm
> -rw-r--r-- 1 madams madams 1482 May 29 09:52 nroff.def
> -rw-r--r-- 1 madams madams   79 May 29 09:52 previous.gif
> -rw-r--r-- 1 madams madams  299 May 29 09:52 previous.xbm
> -rw-r--r-- 1 madams madams 7353 May 29 09:52 pstoppm.ps
> -rw-r--r-- 1 madams madams  326 May 29 09:52 purpleball.gif
> -rw-r--r-- 1 madams madams  326 May 29 09:52 redball.gif
> -rw-r--r-- 1 madams madams  283 May 29 09:52 refman.def
> -rw-r--r-- 1 madams madams 3151 May 29 09:52 refman.sty
> -rw-r--r-- 1 madams madams   74 May 29 09:52 up.gif
> -rw-r--r-- 1 madams madams  281 May 29 09:52 up.xbm
> -rw-r--r-- 1 madams madams  324 May 29 09:52 yellowball.gif
>
>
>
>
> On Fri, May 29, 2015 at 11:23 AM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> I tried building on edison - and can't reproduce this problem.
>>
>> ./configure --COPTFLAGS="-fast -no-ipo" --CXXOPTFLAGS="-fast -no-ipo"
>> --FOPTFLAGS="-fast -no-ipo"  --with-ssl=0 --with-cc=cc
>> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0
>> --with-debugging=0 --with-fc=ftn  --with-fortranlib-autodetect=0
>> --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun  LIBS=-lstdc++
>> --with-64-bit-indices PETSC_DIR=$PWD PETSC_ARCH=arch-xc30-opt64-intel
>> --prefix=$HOME/soft/test
>>
>> The error message you get is puzzling..
>>
>> > Unable to find bfort-base.txt config file in
>> /global/homes/m/madams/petsc_install/petsc/arch-xc30-opt64-intel/share*******
>>
>> Can you attempt a build from a 'clean' repo?
>>
>> rm -rf arch*
>> git clean -f -d -x
>> ./configure....
>>
>> Satish
>>
>>
>> On Fri, 29 May 2015, Mark Adams wrote:
>>
>> > Barry, It looks like you blew it.
>> >
>> > On Wed, May 27, 2015 at 4:03 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> > >
>> > >    Apparently all people with access to the PETSc repository (i.e.
>> > > everyone in the world with internet access) have caught a rare
>> disease that
>> > > has caused their fingers to fall off, making them incapable of fixing
>> this.
>> > >
>> > >    I have added all the PETSc enums to this file in master and next.
>> Make
>> > > sure you run make allfortranstubs to regenerate the stubs.
>> > >
>> > >    Barry
>> > >
>> > >
>> > >
>> > > > On May 27, 2015, at 2:15 PM, Mark Adams <mfadams at lbl.gov> wrote:
>> > > >
>> > > > FYI, this does not seem to be fixed.
>> > > >
>> > > > On Sat, Apr 18, 2015 at 5:49 AM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > > >
>> > > >   Looks like a bunch of these things are missing from
>> > > lib/petsc/conf/bfort-petsc.txt  it should be listed under native
>> > > >
>> > > >   Barry
>> > > >
>> > > >
>> > > >
>> > > > > On Apr 17, 2015, at 4:43 PM, Matthew Knepley <knepley at gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Does bfort have to be told that PCCompositeType is an enum?
>> > > > >
>> > > > >    Matt "bfort's biggest fan"
>> > > > >
>> > > > > On Fri, Apr 17, 2015 at 4:37 PM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > > > > It looks like fieldsplitf.c has an extra PetscToPointer.  type is
>> > > already a pointer:
>> > > > >
>> > > > > PETSC_EXTERN void PETSC_STDCALL  pcfieldsplitgettype_(PC
>> > > pc,PCCompositeType *type, int *__ierr ){
>> > > > > *__ierr = PCFieldSplitGetType(
>> > > > >     (PC)PetscToPointer((pc) ),
>> > > > >     (PCCompositeType* )PetscToPointer((type) ));
>> > > > > }
>> > > > >
>> > > > > On Fri, Apr 17, 2015 at 4:12 PM, Barry Smith <bsmith at mcs.anl.gov>
>> > > wrote:
>> > > > >
>> > > > >    Likely need to run in the debugger to see what generates the
>> error.
>> > > Perhaps a missing ierr?
>> > > > >
>> > > > > Barry
>> > > > >
>> > > > >
>> > > > > > On Apr 17, 2015, at 9:10 AM, Mark Adams <mfadams at lbl.gov>
>> wrote:
>> > > > > >
>> > > > > > I declare
>> > > > > >
>> > > > > >   PCCompositeType::pcctype
>> > > > > >
>> > > > > > But PCFieldSplitGetType does not seem to like it?
>> > > > > >
>> > > > > > Mark
>> > > > > >
>> > > > > > [1]PETSC ERROR: --------------------- Error Message
>> > > --------------------------------------------------------------
>> > > > > > [1]PETSC ERROR: Invalid pointer
>> > > > > > [1]PETSC ERROR: Invalid Pointer to Int: Parameter # 2
>> > > > > > [1]PETSC ERROR: See
>> > > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>> shooting.
>> > > > > > [1]PETSC ERROR: Petsc Development GIT revision:
>> > > v3.5.3-2702-g1f714da  GIT Date: 2015-04-16 11:32:06 -0500
>> > > > > > [1]PETSC ERROR: ../../XGC1_3/xgc-em on a arch-macosx-gnu-g named
>> > > MacBook-Pro.local by markadams Fri Apr 17 10:06:09 2015
>> > > > > > [1]PETSC ERROR: Configure options --download-hypre=1
>> > > --download-p4est --download-superlu=1 --download-mumps
>> --download-scalapack
>> > > --download-superlu_dist=1 --download-parmetis=1 --download-metis=1
>> > > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5
>> > > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g
>> > > > > > [1]PETSC ERROR: #1 PCFieldSplitGetType() line 2046 in
>> > > /Users/markadams/Codes/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > What most experimenters take for granted before they begin their
>> > > experiments is infinitely more interesting than any results to which
>> their
>> > > experiments lead.
>> > > > > -- Norbert Wiener
>> > > >
>> > > >
>> > >
>> > >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/e167d105/attachment.html>

From bsmith at mcs.anl.gov  Fri May 29 13:21:43 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 29 May 2015 13:21:43 -0500
Subject: [petsc-users] SNESVINEWTONRSLS: definition of active set?
In-Reply-To: <CALFwutOr9cGq=KjDCwniyBEBDTh9LMuuZPEqb0r7Vbn=jLa9jA@mail.gmail.com>
References: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>
	<5ABC2C8A-1BE6-489C-BACD-E8D780CD205A@mcs.anl.gov>
	<CALFwutOr9cGq=KjDCwniyBEBDTh9LMuuZPEqb0r7Vbn=jLa9jA@mail.gmail.com>
Message-ID: <9B83FB13-0727-4BB9-A7A8-A14EC1DC8FC0@mcs.anl.gov>


> On May 29, 2015, at 4:19 AM, Asbj?rn Nilsen Riseth <riseth at maths.ox.ac.uk> wrote:
> 
> Barry,
> 
> I changed the code, but it only makes a difference at the order of 1e-10 - so that's not the cause. I've attached (if that's appropriate?) the patch in case anyone is interested. 
> 
> Investigating the function values, I see that the first Newton step goes towards the expected solution for my function values. Then it shoots back close to the initial conditions.

   When does it "shoot back close to the initial conditions"? At the second Newton step? If so is the residual norm still smaller at the second step?

> At the time the solver hits tolerance for the inactive set; the function value is 100-1000 at some of the active set indices.
> Reducing the timestep by an order of magnitude shows the same behavior for the first two timesteps.
> 
> Maybe VI is not the right approach. The company I work with seem to just be projecting negative values.

   The VI solver is essentially just a "more sophisticated" version of "projecting negative values" so should not work worse then an ad hoc method and should generally work better (sometimes much better).

   Is your code something simple you could email me to play with or is it a big application that would be hard for me to experiment with?



  Barry

> 
> I'll investigate further.
> 
> Ozzy
> 
> 
> On Thu, 28 May 2015 at 20:26 Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Ozzy,
> 
>      I cannot say why it is implemented as >=  (could be in error). Just try changing the PETSc code (just run make gnumake in the PETSc directory after you change the source to update the library) and see how it affects your code run.
> 
>    Barry
> 
> > On May 28, 2015, at 3:15 AM, Asbj?rn Nilsen Riseth <riseth at maths.ox.ac.uk> wrote:
> >
> > Dear PETSc developers,
> >
> > Is the active set in NewtonRSLS defined differently from the reference* you give in the documentation on purpose?
> > The reference defines the active set as:
> > x_i = 0 and F_i > 0,
> > whilst the PETSc code defines it as x_i = 0 and F_i >= 0 (vi.c: 356) :
> > !((PetscRealPart(x[i]) > PetscRealPart(xl[i]) + 1.e-8 || (PetscRealPart(f[i]) < 0.0)
> > So PETSc freezes the variables if f[i] == 0.
> >
> > I've been using the Newton RSLS method to ensure positivity in a subsurface flow problem I'm working on. My solution stays almost constant for two timesteps (seemingly independent of the size of the timestep), before it goes towards the expected solution.
> > From my initial conditions, certain variables are frozen because x_i = 0 and f[i] = 0, and I was wondering if that could be the cause of my issue.
> >
> >
> > *:
> > - T. S. Munson, and S. Benson. Flexible Complementarity Solvers for Large-Scale Applications, Optimization Methods and Software, 21 (2006).
> >
> >
> > Regards,
> > Ozzy
> 
> <0001-Define-active-and-inactive-sets-correctly.patch>


From may at bu.edu  Fri May 29 13:33:31 2015
From: may at bu.edu (Young, Matthew, Adam)
Date: Fri, 29 May 2015 18:33:31 +0000
Subject: [petsc-users] Solving Singular Systems
Message-ID: <17A35C213185A84BB8ED54C88FBFD712C3D0C87D@IST-EX10MBX-4.ad.bu.edu>

I have an elliptic PDE of the form Div[n(x) T Grad(phi)]=f(n,...) subject to periodic BC. (phi is the electrostatic potential in a PIC/fluid model). I expect there to be a constant null space but I'm not sure I'm dealing with it properly.

1) Section 4.6 of the manual suggests MatNullSpaceCreate() followed by KSPSetNullSpace(), but I have also seen examples that use MatSetNullSpace(). Is one preferred over the other?
2) Should I follow XXXSetNullSpace() with MatNullSpaceRemove() on the RHS Vec?
3) Do I need to MatNullSpaceDestroy() for either XXXSetNullSpace() option?
4) If so, should the destruction happen after KSPSolve()?
5) The manual (same section) says "You can run with the additional options -pc_factor_shift_nonzero <dampingfactor> or -pc_factor_shift_nonzero <dampingfactor> to prevent a zero pivot." This seems like a typo.

I believe I've done my due Google diligence, but I'm happy to be pointed elsewhere if these questions have come up before.

--A Different Matt

--------------------------------------------------------------
Matthew Young
Graduate Student
Boston University Dept. of Astronomy
--------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/561544e1/attachment.html>

From snakexf at gmail.com  Fri May 29 13:39:05 2015
From: snakexf at gmail.com (Feng Xing)
Date: Fri, 29 May 2015 20:39:05 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
Message-ID: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>

Hello everyone, 

I want to use the the multigrid preconditioner in Petsc/hypre library to a finite element elliptic problem. But there are a lot of parameters. I would to ask if some of them are important for the convergence and the computation time. Thank you very much.

Feng Xing
Postdoc Inria France

From bsmith at mcs.anl.gov  Fri May 29 13:47:27 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 29 May 2015 13:47:27 -0500
Subject: [petsc-users] Solving Singular Systems
In-Reply-To: <17A35C213185A84BB8ED54C88FBFD712C3D0C87D@IST-EX10MBX-4.ad.bu.edu>
References: <17A35C213185A84BB8ED54C88FBFD712C3D0C87D@IST-EX10MBX-4.ad.bu.edu>
Message-ID: <B785C7D1-6EB1-424A-970D-5035C5125B96@mcs.anl.gov>


> On May 29, 2015, at 1:33 PM, Young, Matthew, Adam <may at bu.edu> wrote:
> 
> I have an elliptic PDE of the form Div[n(x) T Grad(phi)]=f(n,...) subject to periodic BC. (phi is the electrostatic potential in a PIC/fluid model). I expect there to be a constant null space but I'm not sure I'm dealing with it properly. 
> 
> 1) Section 4.6 of the manual suggests MatNullSpaceCreate() followed by KSPSetNullSpace(), but I have also seen examples that use MatSetNullSpace(). Is one preferred over the other?

   MatSetNullSpace() is preferred. We are deprecating the KSP version.

> 2) Should I follow XXXSetNullSpace() with MatNullSpaceRemove() on the RHS Vec?

   This will make your linear system consistent so you should probably do it. 

> 3) Do I need to MatNullSpaceDestroy() for either XXXSetNullSpace() option?

   Yes, otherwise you will have a memory leak.
> 4) If so, should the destruction happen after KSPSolve()?

   It doesn't matter when since PETSc does reference counting.


> 5) The manual (same section) says "You can run with the additional options -pc_factor_shift_nonzero <dampingfactor> or -pc_factor_shift_nonzero <dampingfactor> to prevent a zero pivot." This seems like a typo.

   Thanks, looks like it is fixed in the development copy.

   In our next release is a new function MatSetTransposeNullSpace() if you call that also then KSPSolve() will use it to automatically remove from the right hand side the inconsistent part so you do not need to call the MatNullSpaceRemove your self.


> 
> I believe I've done my due Google diligence, but I'm happy to be pointed elsewhere if these questions have come up before.
> 
> --A Different Matt
> 
> --------------------------------------------------------------
> Matthew Young
> Graduate Student
> Boston University Dept. of Astronomy
> --------------------------------------------------------------


From jed at jedbrown.org  Fri May 29 14:17:53 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 29 May 2015 21:17:53 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
In-Reply-To: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
References: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
Message-ID: <87382fjiqm.fsf@jedbrown.org>

Feng Xing <snakexf at gmail.com> writes:
> I want to use the the multigrid preconditioner in Petsc/hypre library
> to a finite element elliptic problem. But there are a lot of
> parameters. I would to ask if some of them are important for the
> convergence and the computation time.

Yes, some of them are important.  We can't comment without further
information on your problem, but custom tuning of solvers for each
problem encountered by each user is not scalable for the PETSc
developers, so we recommend that you consult the Hypre manual (if you're
using Hypre) and experiment with the usual tuning parameters (e.g., the
"strong threshold" discussed in the manual).  When you have more
experience with the problem you're solving and the behavior of the
parameters, feel free to write with more specific questions.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/6d895b15/attachment.pgp>

From snakexf at gmail.com  Fri May 29 14:44:21 2015
From: snakexf at gmail.com (Feng Xing)
Date: Fri, 29 May 2015 21:44:21 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
In-Reply-To: <87382fjiqm.fsf@jedbrown.org>
References: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
	<87382fjiqm.fsf@jedbrown.org>
Message-ID: <8BA50790-4E43-45C7-AF42-F35B3170D606@gmail.com>

Hello,

Thank you for replying so quick.

It?s a variation of 3d Poisson equation ( -Delta u=f, u is unknown, f is a given function ). Mathematically, they have similar properties.  So, I didn?t precise the details to make it look like too complicated. Numerically, the discretisation is by the standard finite element method or the finite volume method.

Regards,

Feng Xing
Postdoc Inria, France


> On 29 May 2015, at 21:17, Jed Brown <jed at jedbrown.org> wrote:
> 
> Feng Xing <snakexf at gmail.com> writes:
>> I want to use the the multigrid preconditioner in Petsc/hypre library
>> to a finite element elliptic problem. But there are a lot of
>> parameters. I would to ask if some of them are important for the
>> convergence and the computation time.
> 
> Yes, some of them are important.  We can't comment without further
> information on your problem, but custom tuning of solvers for each
> problem encountered by each user is not scalable for the PETSc
> developers, so we recommend that you consult the Hypre manual (if you're
> using Hypre) and experiment with the usual tuning parameters (e.g., the
> "strong threshold" discussed in the manual).  When you have more
> experience with the problem you're solving and the behavior of the
> parameters, feel free to write with more specific questions.


From jed at jedbrown.org  Fri May 29 14:55:07 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 29 May 2015 21:55:07 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
In-Reply-To: <8BA50790-4E43-45C7-AF42-F35B3170D606@gmail.com>
References: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
	<87382fjiqm.fsf@jedbrown.org>
	<8BA50790-4E43-45C7-AF42-F35B3170D606@gmail.com>
Message-ID: <87twuvi2g4.fsf@jedbrown.org>

Feng Xing <snakexf at gmail.com> writes:
> It?s a variation of 3d Poisson equation ( -Delta u=f, u is unknown, f
> is a given function ). Mathematically, they have similar properties.

Variable coefficients?  Negative shift?  Geometry?  (These totally
change the behavior of the equation.)  Or really just a Laplacian in a
nice domain?

> So, I didn?t precise the details to make it look like too
> complicated. Numerically, the discretisation is by the standard finite
> element method or the finite volume method.

So this is a test problem and nobody cares about the answer?  You're
using it as a proxy for something else?

It is extremely common to reach misleading conclusions when using the
Laplacian as a test.  If you really care about solving this particular
problem efficiently, there are a myriad of techniques for improving
performance by orders of magnitude.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/2699cf69/attachment.pgp>

From snakexf at gmail.com  Fri May 29 15:22:09 2015
From: snakexf at gmail.com (Feng Xing)
Date: Fri, 29 May 2015 22:22:09 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
In-Reply-To: <87twuvi2g4.fsf@jedbrown.org>
References: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
	<87382fjiqm.fsf@jedbrown.org>
	<8BA50790-4E43-45C7-AF42-F35B3170D606@gmail.com>
	<87twuvi2g4.fsf@jedbrown.org>
Message-ID: <D5D9B928-57D2-4A3A-B9C1-7B7D92D7ECE2@gmail.com>

In detail, the equation is div( K(x) grad(u) ) = f, where K(x) could be a variable coefficient or a constant, in a nice domain ((0,1)^3). The problem is defined on a nice domain like (0,1)^3. Laplace is a special case if K(x)=cst. So I wanted to try to understand it (convergence of multigrid ). 

Regards,

> On 29 May 2015, at 21:55, Jed Brown <jed at jedbrown.org> wrote:
> 
> Feng Xing <snakexf at gmail.com> writes:
>> It?s a variation of 3d Poisson equation ( -Delta u=f, u is unknown, f
>> is a given function ). Mathematically, they have similar properties.
> 
> Variable coefficients?  Negative shift?  Geometry?  (These totally
> change the behavior of the equation.)  Or really just a Laplacian in a
> nice domain?
> 
>> So, I didn?t precise the details to make it look like too
>> complicated. Numerically, the discretisation is by the standard finite
>> element method or the finite volume method.
> 
> So this is a test problem and nobody cares about the answer?  You're
> using it as a proxy for something else?
> 
> It is extremely common to reach misleading conclusions when using the
> Laplacian as a test.  If you really care about solving this particular
> problem efficiently, there are a myriad of techniques for improving
> performance by orders of magnitude.
> myriad

From jed at jedbrown.org  Fri May 29 15:46:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 29 May 2015 22:46:12 +0200
Subject: [petsc-users] Multigrid preconditioner parameters
In-Reply-To: <D5D9B928-57D2-4A3A-B9C1-7B7D92D7ECE2@gmail.com>
References: <EC998D68-21F6-4CA0-B6F8-ACA0FBD65E4C@gmail.com>
	<87382fjiqm.fsf@jedbrown.org>
	<8BA50790-4E43-45C7-AF42-F35B3170D606@gmail.com>
	<87twuvi2g4.fsf@jedbrown.org>
	<D5D9B928-57D2-4A3A-B9C1-7B7D92D7ECE2@gmail.com>
Message-ID: <87k2vri02z.fsf@jedbrown.org>

Feng Xing <snakexf at gmail.com> writes:

> In detail, the equation is div( K(x) grad(u) ) = f, where K(x) could
> be a variable coefficient or a constant, in a nice domain
> ((0,1)^3). The problem is defined on a nice domain like
> (0,1)^3. Laplace is a special case if K(x)=cst. So I wanted to try to
> understand it (convergence of multigrid ).

Be aware that the coefficient structure fundamentally changes the
behavior of the equation and the expected convergence properties.  With
coefficient jumps of 10^10 and fine structure with long correlation
length, the discretization will always be under-resolved.  Depending on
the variation and geometry, it can be easy to solve or very difficult.

One way to understand this: think of a large maze drawn in the plane.
Set Dirichlet u=0 and u=1 boundary conditions at the start and end, with
Neumann on all other surfaces, K(x) = \epsilon on walls, and K(x)=1
everywhere else.  Solve this elliptic problem -- the solution of the
maze just follows the gradient (all corridors leading to dead-ends are
constant).  Obviously this is highly non-local -- small far-away changes
in K(x) completely changes the solution.  Nothing like what you're used
to if you solve smooth Laplacians.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150529/36b9d9d9/attachment.pgp>

From riseth at maths.ox.ac.uk  Fri May 29 19:09:05 2015
From: riseth at maths.ox.ac.uk (=?UTF-8?Q?Asbj=C3=B8rn_Nilsen_Riseth?=)
Date: Sat, 30 May 2015 00:09:05 +0000
Subject: [petsc-users] SNESVINEWTONRSLS: definition of active set?
In-Reply-To: <9B83FB13-0727-4BB9-A7A8-A14EC1DC8FC0@mcs.anl.gov>
References: <CALFwutOKpZijn8K==Ltfyp9qrjc34TJ7XEkAkGoUf0Eg3xBxuw@mail.gmail.com>
	<5ABC2C8A-1BE6-489C-BACD-E8D780CD205A@mcs.anl.gov>
	<CALFwutOr9cGq=KjDCwniyBEBDTh9LMuuZPEqb0r7Vbn=jLa9jA@mail.gmail.com>
	<9B83FB13-0727-4BB9-A7A8-A14EC1DC8FC0@mcs.anl.gov>
Message-ID: <CALFwutMBd79deaMPuxKMBtVmbF0vxFKKGyNxK_5sSYQ8UmikVw@mail.gmail.com>

Hey Barry,

thanks for the offer to have a look at the code. I ran SNESTEST today:
user-defined failed, 1.0 and -1.0 seemed to work fine. My first step will
have to be to find out what's wrong with my jacobian. If I've still got
issues after that, I'll try to set up an easy-to-experiment code

The code is a DG0 FVM formulation set up in Firedrake (a "fork" of FEniCS).
I was assuming UFL would sort the Jacobian for me.
Lesson learnt: always do a SNESTEST.


Ozzy

On Fri, 29 May 2015 at 19:21 Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 29, 2015, at 4:19 AM, Asbj?rn Nilsen Riseth <
> riseth at maths.ox.ac.uk> wrote:
> >
> > Barry,
> >
> > I changed the code, but it only makes a difference at the order of 1e-10
> - so that's not the cause. I've attached (if that's appropriate?) the patch
> in case anyone is interested.
> >
> > Investigating the function values, I see that the first Newton step goes
> towards the expected solution for my function values. Then it shoots back
> close to the initial conditions.
>
>    When does it "shoot back close to the initial conditions"? At the
> second Newton step? If so is the residual norm still smaller at the second
> step?
>
> > At the time the solver hits tolerance for the inactive set; the function
> value is 100-1000 at some of the active set indices.
> > Reducing the timestep by an order of magnitude shows the same behavior
> for the first two timesteps.
> >
> > Maybe VI is not the right approach. The company I work with seem to just
> be projecting negative values.
>
>    The VI solver is essentially just a "more sophisticated" version of
> "projecting negative values" so should not work worse then an ad hoc method
> and should generally work better (sometimes much better).
>
>    Is your code something simple you could email me to play with or is it
> a big application that would be hard for me to experiment with?
>
>
>
>   Barry
>
> >
> > I'll investigate further.
> >
> > Ozzy
> >
> >
> > On Thu, 28 May 2015 at 20:26 Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Ozzy,
> >
> >      I cannot say why it is implemented as >=  (could be in error). Just
> try changing the PETSc code (just run make gnumake in the PETSc directory
> after you change the source to update the library) and see how it affects
> your code run.
> >
> >    Barry
> >
> > > On May 28, 2015, at 3:15 AM, Asbj?rn Nilsen Riseth <
> riseth at maths.ox.ac.uk> wrote:
> > >
> > > Dear PETSc developers,
> > >
> > > Is the active set in NewtonRSLS defined differently from the
> reference* you give in the documentation on purpose?
> > > The reference defines the active set as:
> > > x_i = 0 and F_i > 0,
> > > whilst the PETSc code defines it as x_i = 0 and F_i >= 0 (vi.c: 356) :
> > > !((PetscRealPart(x[i]) > PetscRealPart(xl[i]) + 1.e-8 ||
> (PetscRealPart(f[i]) < 0.0)
> > > So PETSc freezes the variables if f[i] == 0.
> > >
> > > I've been using the Newton RSLS method to ensure positivity in a
> subsurface flow problem I'm working on. My solution stays almost constant
> for two timesteps (seemingly independent of the size of the timestep),
> before it goes towards the expected solution.
> > > From my initial conditions, certain variables are frozen because x_i =
> 0 and f[i] = 0, and I was wondering if that could be the cause of my issue.
> > >
> > >
> > > *:
> > > - T. S. Munson, and S. Benson. Flexible Complementarity Solvers for
> Large-Scale Applications, Optimization Methods and Software, 21 (2006).
> > >
> > >
> > > Regards,
> > > Ozzy
> >
> > <0001-Define-active-and-inactive-sets-correctly.patch>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/cdbe4c75/attachment-0001.html>

From jychang48 at gmail.com  Sat May 30 02:37:45 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Sat, 30 May 2015 02:37:45 -0500
Subject: [petsc-users] Modified Taylor-Hood elements with piece-wise
 constant pressure for Stokes equation
Message-ID: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>

Hello everyone,

In SNES ex62, it seems to solve the Stokes equation using TH elements
(i.e., using P2-P1 elements). However, this formulation is not locally
conservative, and as a result many people use the P2-P1/P0 elements where
the pressure space is enriched with a piece-wise constant to ensure mass
conservation at the element level.

Is such an element combination possible with the current implementation
within DMPlex?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/d5d4f556/attachment.html>

From rongliang.chan at gmail.com  Sat May 30 05:05:31 2015
From: rongliang.chan at gmail.com (Rongliang Chen)
Date: Sat, 30 May 2015 18:05:31 +0800
Subject: [petsc-users] NASM FormFunctionLocal
Message-ID: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>

Hi there,

I am trying to use NASM to solve a nonlinear problem. In my
FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the
"-da_overlap 0" case, the code works fine.

For the "-da_overlap 1" case, when I use the vector "myvec" created by
DMCreateLocalVector, it works fine. But if I use the vector "myvec" created
by DMCreateGlobalVector, I got the error messages:
-----------------------
[6]Arguments are incompatible
[6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local
sizes 619528 664392
----------------------

And when I switch to "-da_overlap 2", both of the global and local version
of "mrvec" got error messages:
----------------------------------
[10]PETSC ERROR: Arguments are incompatible
[10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local
sizes 633144 678680

[3]PETSC ERROR: Arguments are incompatible
[3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local
sizes 650256 696540
-----------------------------------

Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for
NASM? Thanks.

Best regards,
Rongliang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/95f3c7b3/attachment.html>

From knepley at gmail.com  Sat May 30 09:55:42 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 30 May 2015 09:55:42 -0500
Subject: [petsc-users] NASM FormFunctionLocal
In-Reply-To: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
References: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
Message-ID: <CAMYG4GmceYNTQnxFNTifPJ+5O39x3Rq1sMuY7mgDvt4WvJjiWA@mail.gmail.com>

On Sat, May 30, 2015 at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com>
wrote:

> Hi there,
>
> I am trying to use NASM to solve a nonlinear problem. In my
> FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the
> "-da_overlap 0" case, the code works fine.
>
> For the "-da_overlap 1" case, when I use the vector "myvec" created by
> DMCreateLocalVector, it works fine. But if I use the vector "myvec" created
> by DMCreateGlobalVector, I got the error messages:
> -----------------------
> [6]Arguments are incompatible
> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local
> sizes 619528 664392
> ----------------------
>
> And when I switch to "-da_overlap 2", both of the global and local version
> of "mrvec" got error messages:
> ----------------------------------
> [10]PETSC ERROR: Arguments are incompatible
> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA
> local sizes 633144 678680
>
> [3]PETSC ERROR: Arguments are incompatible
> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local
> sizes 650256 696540
> -----------------------------------
>
> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for
> NASM? Thanks.
>

1) You have to show the full error messages

2) You should simplify the problem for debugging. Run on 1 or 2 processes

3) The vector that comes into FormFunctionLocal() is a local vector

  Thanks,

    Matt


> Best regards,
> Rongliang
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/b4245bed/attachment.html>

From knepley at gmail.com  Sat May 30 10:08:08 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 30 May 2015 10:08:08 -0500
Subject: [petsc-users] Modified Taylor-Hood elements with piece-wise
 constant pressure for Stokes equation
In-Reply-To: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
References: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
Message-ID: <CAMYG4Gmjum1+eSCi7ndyXP1WOk0j7eCDd=ELLHrggm-yCLiq9g@mail.gmail.com>

On Sat, May 30, 2015 at 2:37 AM, Justin Chang <jychang48 at gmail.com> wrote:

> Hello everyone,
>
> In SNES ex62, it seems to solve the Stokes equation using TH elements
> (i.e., using P2-P1 elements). However, this formulation is not locally
> conservative, and as a result many people use the P2-P1/P0 elements where
> the pressure space is enriched with a piece-wise constant to ensure mass
> conservation at the element level.
>
> Is such an element combination possible with the current implementation
> within DMPlex?
>

I have implemented P1_disc:


https://bitbucket.org/petsc/petsc/src/779682a0c231963f14b692ce9dcbb6520a47bf75/config/builder.py?at=master#cl-497

I believe its working correctly based on my tests for function
interpolation, but I have not tried anything hard with it.

 Thanks,

   Matt


> Thanks,
> Justin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/230a44db/attachment.html>

From bsmith at mcs.anl.gov  Sat May 30 12:09:06 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 30 May 2015 12:09:06 -0500
Subject: [petsc-users] NASM FormFunctionLocal
In-Reply-To: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
References: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
Message-ID: <066427A9-8A57-487A-AA9C-7583F78C7279@mcs.anl.gov>


> On May 30, 2015, at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
> 
> Hi there,
> 
> I am trying to use NASM to solve a nonlinear problem. In my FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the "-da_overlap 0" case, the code works fine. 
> 
> For the "-da_overlap 1" case, when I use the vector "myvec" created by DMCreateLocalVector, it works fine. But if I use the vector "myvec" created by DMCreateGlobalVector, I got the error messages:
> -----------------------
> [6]Arguments are incompatible
> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local sizes 619528 664392

I think you are changing something else at the same time you changed the overlap.  The number ^^^^  619528 is the local size of the global vector and it should remain the same when you change the overlap but below you have TWO other numbers 633144 and 650256  Do you have more than one DMDA? 
Run changing absolutely nothing except the overlap.
> ----------------------
> 
> And when I switch to "-da_overlap 2", both of the global and local version of "mrvec" got error messages: 
> ----------------------------------
> [10]PETSC ERROR: Arguments are incompatible
> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local sizes 633144 678680
> 
> [3]PETSC ERROR: Arguments are incompatible
> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local sizes 650256 696540
> -----------------------------------
> 
> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for NASM? Thanks.
> 
> Best regards,
> Rongliang
> 


From fdkong.jd at gmail.com  Sat May 30 12:17:30 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Sat, 30 May 2015 11:17:30 -0600
Subject: [petsc-users] petsc-users Digest, Vol 77, Issue 108
In-Reply-To: <mailman.55.1433005208.1127.petsc-users@mcs.anl.gov>
References: <mailman.55.1433005208.1127.petsc-users@mcs.anl.gov>
Message-ID: <CAN5Wd-+3+zWn3Wx1tFVQe7oH8-BjLZ85m+i6gZR7Ctu9W6ZBtQ@mail.gmail.com>

Message: 2
> Date: Sat, 30 May 2015 18:05:31 +0800
> From: Rongliang Chen <rongliang.chan at gmail.com>
> To: PETSc Users mail list <petsc-users at mcs.anl.gov>
> Subject: [petsc-users] NASM FormFunctionLocal
> Message-ID:
>         <
> CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi there,
>
> I am trying to use NASM to solve a nonlinear problem. In my
> FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the
> "-da_overlap 0" case, the code works fine.
>
> For the "-da_overlap 1" case, when I use the vector "myvec" created by
> DMCreateLocalVector, it works fine. But if I use the vector "myvec" created
> by DMCreateGlobalVector, I got the error messages:
> -----------------------
> [6]Arguments are incompatible
> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local
> sizes 619528 664392
> ----------------------
>
> And when I switch to "-da_overlap 2", both of the global and local version
> of "mrvec" got error messages:
> ----------------------------------
> [10]PETSC ERROR: Arguments are incompatible
> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local
> sizes 633144 678680
>
> [3]PETSC ERROR: Arguments are incompatible
> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local
> sizes 650256 696540
> -----------------------------------
>
> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for
> NASM? Thanks.
>
> Best regards,
> Rongliang
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/95f3c7b3/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 3
> Date: Sat, 30 May 2015 09:55:42 -0500
> From: Matthew Knepley <knepley at gmail.com>
> To: Rongliang Chen <rongliang.chan at gmail.com>
> Cc: PETSc Users mail list <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] NASM FormFunctionLocal
> Message-ID:
>         <
> CAMYG4GmceYNTQnxFNTifPJ+5O39x3Rq1sMuY7mgDvt4WvJjiWA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Sat, May 30, 2015 at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com>
> wrote:
>
> > Hi there,
> >
> > I am trying to use NASM to solve a nonlinear problem. In my
> > FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the
> > "-da_overlap 0" case, the code works fine.
> >
> > For the "-da_overlap 1" case, when I use the vector "myvec" created by
> > DMCreateLocalVector, it works fine. But if I use the vector "myvec"
> created
> > by DMCreateGlobalVector, I got the error messages:
> > -----------------------
> > [6]Arguments are incompatible
> > [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA
> local
> > sizes 619528 664392
> > ----------------------
> >
> > And when I switch to "-da_overlap 2", both of the global and local
> version
> > of "mrvec" got error messages:
> > ----------------------------------
> > [10]PETSC ERROR: Arguments are incompatible
> > [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA
> > local sizes 633144 678680
> >
> > [3]PETSC ERROR: Arguments are incompatible
> > [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA
> local
> > sizes 650256 696540
> > -----------------------------------
> >
> > Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for
> > NASM? Thanks.
> >
>
> 1) You have to show the full error messages
>
> 2) You should simplify the problem for debugging. Run on 1 or 2 processes
>
> 3) The vector that comes into FormFunctionLocal() is a local vector
>
>   Thanks,
>
>     Matt
>
>
> > Best regards,
> > Rongliang
> >
> >
>


To use  the FormFunctionLocal, your input has to be local vector. If you
want to use the global vector as your input, you can add one function to
scatter global vector to local vector.


Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/c7df97f9/attachment.html>

From rongliang.chan at gmail.com  Sat May 30 20:02:39 2015
From: rongliang.chan at gmail.com (Rongliang Chen)
Date: Sun, 31 May 2015 09:02:39 +0800
Subject: [petsc-users] NASM FormFunctionLocal
In-Reply-To: <066427A9-8A57-487A-AA9C-7583F78C7279@mcs.anl.gov>
References: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
	<066427A9-8A57-487A-AA9C-7583F78C7279@mcs.anl.gov>
Message-ID: <556A5DAF.7040700@gmail.com>

Hi Barry and Matt,

Many thanks for your reply. I only have one DA and I do not think I 
changed others when I changed the overlap.

This error can be reproduced using the attached code (added some lines 
in the src/snes/examples/tutorials/ex19.c) run with:

runex19_1:
         -@${MPIEXEC} -n 2 ./ex19 -da_refine 3 -snes_type nasm 
-da_overlap 2 -snes_monitor

The full error messages are followed:
------------------
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
   0 SNES Function norm 4.066115181565e-02
[1]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[1]PETSC ERROR: Arguments are incompatible
[1]PETSC ERROR: Vector local size 1300 is not compatible with DMDA local 
sizes 1400 1728

[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.2, unknown
[1]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 
31 08:55:16 2015
[1]PETSC ERROR: Configure options --download-fblaslapack 
--download-blacs --download-scalapack --download-metis 
--download-parmetis --download-exodusii --download-netcdf 
--download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 
--with-debugging=1 --with-shared-libraries=0
[1]PETSC ERROR: #1 DMDAVecGetArray() line 71 in 
/home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
[1]PETSC ERROR: #2 FormFunctionLocal() line 275 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
[1]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
[1]PETSC ERROR: #4 SNESComputeFunction() line 2033 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[1]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
[1]PETSC ERROR: #6 SNESSolve() line 3743 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[1]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
[1]PETSC ERROR: #8 SNESSolve_NASM() line 859 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
[1]PETSC ERROR: #9 SNESSolve() line 3743 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[1]PETSC ERROR: #10 main() line 162 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
[1]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 75) - process 1
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Arguments are incompatible
[0]PETSC ERROR: Vector local size 1400 is not compatible with DMDA local 
sizes 1500 1836

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.2, unknown
[0]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 
31 08:55:16 2015
[0]PETSC ERROR: Configure options --download-fblaslapack 
--download-blacs --download-scalapack --download-metis 
--download-parmetis --download-exodusii --download-netcdf 
--download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 
--with-debugging=1 --with-shared-libraries=0
[0]PETSC ERROR: #1 DMDAVecGetArray() line 71 in 
/home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
[0]PETSC ERROR: #2 FormFunctionLocal() line 275 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
[0]PETSC ERROR: #4 SNESComputeFunction() line 2033 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 3743 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[0]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
[0]PETSC ERROR: #8 SNESSolve_NASM() line 859 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
[0]PETSC ERROR: #9 SNESSolve() line 3743 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[0]PETSC ERROR: #10 main() line 162 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
[0]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 75) - process 0

=====================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 19200
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
make: [runex19_1] Error 75 (ignored)
-----------------------

Best regards,
Rongliang

On 05/31/2015 01:09 AM, Barry Smith wrote:
>> On May 30, 2015, at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>
>> Hi there,
>>
>> I am trying to use NASM to solve a nonlinear problem. In my FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the "-da_overlap 0" case, the code works fine.
>>
>> For the "-da_overlap 1" case, when I use the vector "myvec" created by DMCreateLocalVector, it works fine. But if I use the vector "myvec" created by DMCreateGlobalVector, I got the error messages:
>> -----------------------
>> [6]Arguments are incompatible
>> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local sizes 619528 664392
> I think you are changing something else at the same time you changed the overlap.  The number ^^^^  619528 is the local size of the global vector and it should remain the same when you change the overlap but below you have TWO other numbers 633144 and 650256  Do you have more than one DMDA?
> Run changing absolutely nothing except the overlap.
>> ----------------------
>>
>> And when I switch to "-da_overlap 2", both of the global and local version of "mrvec" got error messages:
>> ----------------------------------
>> [10]PETSC ERROR: Arguments are incompatible
>> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local sizes 633144 678680
>>
>> [3]PETSC ERROR: Arguments are incompatible
>> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local sizes 650256 696540
>> -----------------------------------
>>
>> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for NASM? Thanks.
>>
>> Best regards,
>> Rongliang
>>

-------------- next part --------------

static char help[] = "Nonlinear driven cavity with multigrid in 2d.\n\
  \n\
The 2D driven cavity problem is solved in a velocity-vorticity formulation.\n\
The flow can be driven with the lid or with bouyancy or both:\n\
  -lidvelocity <lid>, where <lid> = dimensionless velocity of lid\n\
  -grashof <gr>, where <gr> = dimensionless temperature gradent\n\
  -prandtl <pr>, where <pr> = dimensionless thermal/momentum diffusity ratio\n\
  -contours : draw contour plots of solution\n\n";
/*
      See src/ksp/ksp/examples/tutorials/ex45.c
*/

/*T
   Concepts: SNES^solving a system of nonlinear equations (parallel multicomponent example);
   Concepts: DMDA^using distributed arrays;
   Concepts: multicomponent
   Processors: n
T*/
/*F-----------------------------------------------------------------------

    We thank David E. Keyes for contributing the driven cavity discretization within this example code.

    This problem is modeled by the partial differential equation system

\begin{eqnarray}
        - \triangle U - \nabla_y \Omega & = & 0  \\
        - \triangle V + \nabla_x\Omega & = & 0  \\
        - \triangle \Omega + \nabla \cdot ([U*\Omega,V*\Omega]) - GR* \nabla_x T & = & 0  \\
        - \triangle T + PR* \nabla \cdot ([U*T,V*T]) & = & 0
\end{eqnarray}

    in the unit square, which is uniformly discretized in each of x and y in this simple encoding.

    No-slip, rigid-wall Dirichlet conditions are used for $ [U,V]$.
    Dirichlet conditions are used for Omega, based on the definition of
    vorticity: $ \Omega = - \nabla_y U + \nabla_x V$, where along each
    constant coordinate boundary, the tangential derivative is zero.
    Dirichlet conditions are used for T on the left and right walls,
    and insulation homogeneous Neumann conditions are used for T on
    the top and bottom walls.

    A finite difference approximation with the usual 5-point stencil
    is used to discretize the boundary value problem to obtain a
    nonlinear system of equations.  Upwinding is used for the divergence
    (convective) terms and central for the gradient (source) terms.

    The Jacobian can be either
      * formed via finite differencing using coloring (the default), or
      * applied matrix-free via the option -snes_mf
        (for larger grid problems this variant may not converge
        without a preconditioner due to ill-conditioning).

  ------------------------------------------------------------------------F*/

/*
   Include "petscdmda.h" so that we can use distributed arrays (DMDAs).
   Include "petscsnes.h" so that we can use SNES solvers.  Note that this
   file automatically includes:
     petscsys.h       - base PETSc routines   petscvec.h - vectors
     petscmat.h - matrices
     petscis.h     - index sets            petscksp.h - Krylov subspace methods
     petscviewer.h - viewers               petscpc.h  - preconditioners
     petscksp.h   - linear solvers
*/
#if defined(PETSC_APPLE_FRAMEWORK)
#import <PETSc/petscsnes.h>
#import <PETSc/petscdmda.h>
#else
#include <petscsnes.h>
#include <petscdm.h>
#include <petscdmda.h>
#endif

/*
   User-defined routines and data structures
*/
typedef struct {
  PetscScalar u,v,omega,temp;
} Field;

PetscErrorCode FormFunctionLocal(DMDALocalInfo*,Field**,Field**,void*);

typedef struct {
  PassiveReal lidvelocity,prandtl,grashof;  /* physical parameters */
  PetscBool   draw_contours;                /* flag - 1 indicates drawing contours */
  Vec temp; // added by rlchen
} AppCtx;

extern PetscErrorCode FormInitialGuess(AppCtx*,DM,Vec);
extern PetscErrorCode NonlinearGS(SNES,Vec,Vec,void*);

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  AppCtx         user;                /* user-defined work context */
  PetscInt       mx,my,its;
  PetscErrorCode ierr;
  MPI_Comm       comm;
  SNES           snes;
  DM             da;
  Vec            x;

  ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return(1);

  PetscFunctionBeginUser;
  comm = PETSC_COMM_WORLD;
  ierr = SNESCreate(comm,&snes);CHKERRQ(ierr);

  /*
      Create distributed array object to manage parallel grid and vectors
      for principal unknowns (x) and governing residuals (f)
  */
  ierr = DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,-4,-4,PETSC_DECIDE,PETSC_DECIDE,4,1,0,0,&da);CHKERRQ(ierr);
  ierr = SNESSetDM(snes,(DM)da);CHKERRQ(ierr);
  ierr = SNESSetNGS(snes, NonlinearGS, (void*)&user);CHKERRQ(ierr);

  ierr = DMDAGetInfo(da,0,&mx,&my,PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE,
                     PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE,PETSC_IGNORE);CHKERRQ(ierr);
  /*
     Problem parameters (velocity of lid, prandtl, and grashof numbers)
  */
  user.lidvelocity = 1.0/(mx*my);
  user.prandtl     = 1.0;
  user.grashof     = 1.0;

  ierr = PetscOptionsGetReal(NULL,"-lidvelocity",&user.lidvelocity,NULL);CHKERRQ(ierr);
  ierr = PetscOptionsGetReal(NULL,"-prandtl",&user.prandtl,NULL);CHKERRQ(ierr);
  ierr = PetscOptionsGetReal(NULL,"-grashof",&user.grashof,NULL);CHKERRQ(ierr);
  ierr = PetscOptionsHasName(NULL,"-contours",&user.draw_contours);CHKERRQ(ierr);

  ierr = DMDASetFieldName(da,0,"x_velocity");CHKERRQ(ierr);
  ierr = DMDASetFieldName(da,1,"y_velocity");CHKERRQ(ierr);
  ierr = DMDASetFieldName(da,2,"Omega");CHKERRQ(ierr);
  ierr = DMDASetFieldName(da,3,"temperature");CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     Create user context, set problem data, create vector data structures.
     Also, compute the initial guess.
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     Create nonlinear solver context

     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  ierr = DMCreateLocalVector(da, &user.temp);CHKERRQ(ierr); // added by rlchen
  ierr = VecZeroEntries(user.temp);CHKERRQ(ierr); // added by rlchen

  ierr = DMSetApplicationContext(da,&user);CHKERRQ(ierr);
  ierr = DMDASNESSetFunctionLocal(da,INSERT_VALUES,(PetscErrorCode (*)(DMDALocalInfo*,void*,void*,void*))FormFunctionLocal,&user);CHKERRQ(ierr);
  ierr = SNESSetFromOptions(snes);CHKERRQ(ierr);
  ierr = PetscPrintf(comm,"lid velocity = %g, prandtl # = %g, grashof # = %g\n",(double)user.lidvelocity,(double)user.prandtl,(double)user.grashof);CHKERRQ(ierr);


  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     Solve the nonlinear system
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  ierr = DMCreateGlobalVector(da,&x);CHKERRQ(ierr);
  ierr = FormInitialGuess(&user,da,x);CHKERRQ(ierr);

  ierr = SNESSolve(snes,NULL,x);CHKERRQ(ierr);

  ierr = SNESGetIterationNumber(snes,&its);CHKERRQ(ierr);
  ierr = PetscPrintf(comm,"Number of SNES iterations = %D\n", its);CHKERRQ(ierr);

  /*
     Visualize solution
  */
  if (user.draw_contours) {
    ierr = VecView(x,PETSC_VIEWER_DRAW_WORLD);CHKERRQ(ierr);
  }

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     Free work space.  All PETSc objects should be destroyed when they
     are no longer needed.
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  ierr = VecDestroy(&x);CHKERRQ(ierr);
  ierr = VecDestroy(&user.temp);CHKERRQ(ierr); // added by rlchen
  ierr = DMDestroy(&da);CHKERRQ(ierr);
  ierr = SNESDestroy(&snes);CHKERRQ(ierr);
  ierr = PetscFinalize();
  return 0;
}

/* ------------------------------------------------------------------- */


#undef __FUNCT__
#define __FUNCT__ "FormInitialGuess"
/*
   FormInitialGuess - Forms initial approximation.

   Input Parameters:
   user - user-defined application context
   X - vector

   Output Parameter:
   X - vector
*/
PetscErrorCode FormInitialGuess(AppCtx *user,DM da,Vec X)
{
  PetscInt       i,j,mx,xs,ys,xm,ym;
  PetscErrorCode ierr;
  PetscReal      grashof,dx;
  Field          **x;

  grashof = user->grashof;

  ierr = DMDAGetInfo(da,0,&mx,0,0,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);
  dx   = 1.0/(mx-1);

  /*
     Get local grid boundaries (for 2-dimensional DMDA):
       xs, ys   - starting grid indices (no ghost points)
       xm, ym   - widths of local grid (no ghost points)
  */
  ierr = DMDAGetCorners(da,&xs,&ys,NULL,&xm,&ym,NULL);CHKERRQ(ierr);

  /*
     Get a pointer to vector data.
       - For default PETSc vectors, VecGetArray() returns a pointer to
         the data array.  Otherwise, the routine is implementation dependent.
       - You MUST call VecRestoreArray() when you no longer need access to
         the array.
  */
  ierr = DMDAVecGetArray(da,X,&x);CHKERRQ(ierr);

  /*
     Compute initial guess over the locally owned part of the grid
     Initial condition is motionless fluid and equilibrium temperature
  */
  for (j=ys; j<ys+ym; j++) {
    for (i=xs; i<xs+xm; i++) {
      x[j][i].u     = 0.0;
      x[j][i].v     = 0.0;
      x[j][i].omega = 0.0;
      x[j][i].temp  = (grashof>0)*i*dx;
    }
  }

  /*
     Restore vector
  */
  ierr = DMDAVecRestoreArray(da,X,&x);CHKERRQ(ierr);
  return 0;
}

#undef __FUNCT__
#define __FUNCT__ "FormFunctionLocal"
PetscErrorCode FormFunctionLocal(DMDALocalInfo *info,Field **x,Field **f,void *ptr)
{
  AppCtx         *user = (AppCtx*)ptr;
  PetscErrorCode ierr;
  PetscInt       xints,xinte,yints,yinte,i,j;
  PetscReal      hx,hy,dhx,dhy,hxdhy,hydhx;
  PetscReal      grashof,prandtl,lid;
  PetscScalar    u,uxx,uyy,vx,vy,avx,avy,vxp,vxm,vyp,vym;

  PetscFunctionBeginUser;
  grashof = user->grashof;
  prandtl = user->prandtl;
  lid     = user->lidvelocity;

  /*
     Define mesh intervals ratios for uniform grid.

     Note: FD formulae below are normalized by multiplying through by
     local volume element (i.e. hx*hy) to obtain coefficients O(1) in two dimensions.


  */

    PetscScalar ***z; // added by rlchen
    ierr = DMDAVecGetArray(info->da,user->temp, &z);CHKERRQ(ierr); // added by rlchen
    ierr = DMDAVecRestoreArray(info->da,user->temp,&z);CHKERRQ(ierr); // added by rlchen

  dhx   = (PetscReal)(info->mx-1);  dhy = (PetscReal)(info->my-1);
  hx    = 1.0/dhx;                   hy = 1.0/dhy;
  hxdhy = hx*dhy;                 hydhx = hy*dhx;

  xints = info->xs; xinte = info->xs+info->xm; yints = info->ys; yinte = info->ys+info->ym;

  /* Test whether we are on the bottom edge of the global array */
  if (yints == 0) {
    j     = 0;
    yints = yints + 1;
    /* bottom edge */
    for (i=info->xs; i<info->xs+info->xm; i++) {
      f[j][i].u     = x[j][i].u;
      f[j][i].v     = x[j][i].v;
      f[j][i].omega = x[j][i].omega + (x[j+1][i].u - x[j][i].u)*dhy;
      f[j][i].temp  = x[j][i].temp-x[j+1][i].temp;
    }
  }

  /* Test whether we are on the top edge of the global array */
  if (yinte == info->my) {
    j     = info->my - 1;
    yinte = yinte - 1;
    /* top edge */
    for (i=info->xs; i<info->xs+info->xm; i++) {
      f[j][i].u     = x[j][i].u - lid;
      f[j][i].v     = x[j][i].v;
      f[j][i].omega = x[j][i].omega + (x[j][i].u - x[j-1][i].u)*dhy;
      f[j][i].temp  = x[j][i].temp-x[j-1][i].temp;
    }
  }

  /* Test whether we are on the left edge of the global array */
  if (xints == 0) {
    i     = 0;
    xints = xints + 1;
    /* left edge */
    for (j=info->ys; j<info->ys+info->ym; j++) {
      f[j][i].u     = x[j][i].u;
      f[j][i].v     = x[j][i].v;
      f[j][i].omega = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx;
      f[j][i].temp  = x[j][i].temp;
    }
  }

  /* Test whether we are on the right edge of the global array */
  if (xinte == info->mx) {
    i     = info->mx - 1;
    xinte = xinte - 1;
    /* right edge */
    for (j=info->ys; j<info->ys+info->ym; j++) {
      f[j][i].u     = x[j][i].u;
      f[j][i].v     = x[j][i].v;
      f[j][i].omega = x[j][i].omega - (x[j][i].v - x[j][i-1].v)*dhx;
      f[j][i].temp  = x[j][i].temp - (PetscReal)(grashof>0);
    }
  }

  /* Compute over the interior points */
  for (j=yints; j<yinte; j++) {
    for (i=xints; i<xinte; i++) {

      /*
       convective coefficients for upwinding
      */
      vx  = x[j][i].u; avx = PetscAbsScalar(vx);
      vxp = .5*(vx+avx); vxm = .5*(vx-avx);
      vy  = x[j][i].v; avy = PetscAbsScalar(vy);
      vyp = .5*(vy+avy); vym = .5*(vy-avy);

      /* U velocity */
      u         = x[j][i].u;
      uxx       = (2.0*u - x[j][i-1].u - x[j][i+1].u)*hydhx;
      uyy       = (2.0*u - x[j-1][i].u - x[j+1][i].u)*hxdhy;
      f[j][i].u = uxx + uyy - .5*(x[j+1][i].omega-x[j-1][i].omega)*hx;

      /* V velocity */
      u         = x[j][i].v;
      uxx       = (2.0*u - x[j][i-1].v - x[j][i+1].v)*hydhx;
      uyy       = (2.0*u - x[j-1][i].v - x[j+1][i].v)*hxdhy;
      f[j][i].v = uxx + uyy + .5*(x[j][i+1].omega-x[j][i-1].omega)*hy;

      /* Omega */
      u             = x[j][i].omega;
      uxx           = (2.0*u - x[j][i-1].omega - x[j][i+1].omega)*hydhx;
      uyy           = (2.0*u - x[j-1][i].omega - x[j+1][i].omega)*hxdhy;
      f[j][i].omega = uxx + uyy + (vxp*(u - x[j][i-1].omega) + vxm*(x[j][i+1].omega - u))*hy +
                      (vyp*(u - x[j-1][i].omega) + vym*(x[j+1][i].omega - u))*hx -
                      .5*grashof*(x[j][i+1].temp - x[j][i-1].temp)*hy;

      /* Temperature */
      u            = x[j][i].temp;
      uxx          = (2.0*u - x[j][i-1].temp - x[j][i+1].temp)*hydhx;
      uyy          = (2.0*u - x[j-1][i].temp - x[j+1][i].temp)*hxdhy;
      f[j][i].temp =  uxx + uyy  + prandtl*((vxp*(u - x[j][i-1].temp) + vxm*(x[j][i+1].temp - u))*hy +
                                            (vyp*(u - x[j-1][i].temp) + vym*(x[j+1][i].temp - u))*hx);
    }
  }

  /*
     Flop count (multiply-adds are counted as 2 operations)
  */
  ierr = PetscLogFlops(84.0*info->ym*info->xm);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}


#undef __FUNCT__
#define __FUNCT__ "NonlinearGS"
PetscErrorCode NonlinearGS(SNES snes, Vec X, Vec B, void *ctx)
{
  DMDALocalInfo  info;
  Field          **x,**b;
  PetscErrorCode ierr;
  Vec            localX, localB;
  DM             da;
  PetscInt       xints,xinte,yints,yinte,i,j,k,l;
  PetscInt       max_its,tot_its;
  PetscInt       sweeps;
  PetscReal      rtol,atol,stol;
  PetscReal      hx,hy,dhx,dhy,hxdhy,hydhx;
  PetscReal      grashof,prandtl,lid;
  PetscScalar    u,uxx,uyy,vx,vy,avx,avy,vxp,vxm,vyp,vym;
  PetscScalar    fu, fv, fomega, ftemp;
  PetscScalar    dfudu;
  PetscScalar    dfvdv;
  PetscScalar    dfodu, dfodv, dfodo;
  PetscScalar    dftdu, dftdv, dftdt;
  PetscScalar    yu=0, yv=0, yo=0, yt=0;
  PetscScalar    bjiu, bjiv, bjiomega, bjitemp;
  PetscBool      ptconverged;
  PetscReal      pfnorm,pfnorm0,pynorm,pxnorm;
  AppCtx         *user = (AppCtx*)ctx;

  PetscFunctionBeginUser;
  grashof = user->grashof;
  prandtl = user->prandtl;
  lid     = user->lidvelocity;
  tot_its = 0;
  ierr    = SNESNGSGetTolerances(snes,&rtol,&atol,&stol,&max_its);CHKERRQ(ierr);
  ierr    = SNESNGSGetSweeps(snes,&sweeps);CHKERRQ(ierr);
  ierr    = SNESGetDM(snes,(DM*)&da);CHKERRQ(ierr);
  ierr    = DMGetLocalVector(da,&localX);CHKERRQ(ierr);
  if (B) {
    ierr = DMGetLocalVector(da,&localB);CHKERRQ(ierr);
  }
  /*
     Scatter ghost points to local vector, using the 2-step process
        DMGlobalToLocalBegin(), DMGlobalToLocalEnd().
  */
  ierr = DMGlobalToLocalBegin(da,X,INSERT_VALUES,localX);CHKERRQ(ierr);
  ierr = DMGlobalToLocalEnd(da,X,INSERT_VALUES,localX);CHKERRQ(ierr);
  if (B) {
    ierr = DMGlobalToLocalBegin(da,B,INSERT_VALUES,localB);CHKERRQ(ierr);
    ierr = DMGlobalToLocalEnd(da,B,INSERT_VALUES,localB);CHKERRQ(ierr);
  }
  ierr = DMDAGetLocalInfo(da,&info);CHKERRQ(ierr);
  ierr = DMDAVecGetArray(da,localX,&x);CHKERRQ(ierr);
  if (B) {
    ierr = DMDAVecGetArray(da,localB,&b);CHKERRQ(ierr);
  }
  /* looks like a combination of the formfunction / formjacobian routines */
  dhx   = (PetscReal)(info.mx-1);dhy   = (PetscReal)(info.my-1);
  hx    = 1.0/dhx;               hy    = 1.0/dhy;
  hxdhy = hx*dhy;                hydhx = hy*dhx;

  xints = info.xs; xinte = info.xs+info.xm; yints = info.ys; yinte = info.ys+info.ym;

  /* Set the boundary conditions on the momentum equations */
  /* Test whether we are on the bottom edge of the global array */
  if (yints == 0) {
    j     = 0;
    yints = yints + 1;
    /* bottom edge */
    for (i=info.xs; i<info.xs+info.xm; i++) {

      if (B) {
        bjiu = b[j][i].u;
        bjiv = b[j][i].v;
      } else {
        bjiu = 0.0;
        bjiv = 0.0;
      }
      x[j][i].u = 0.0 + bjiu;
      x[j][i].v = 0.0 + bjiv;
    }
  }

  /* Test whether we are on the top edge of the global array */
  if (yinte == info.my) {
    j     = info.my - 1;
    yinte = yinte - 1;
    /* top edge */
    for (i=info.xs; i<info.xs+info.xm; i++) {
      if (B) {
        bjiu = b[j][i].u;
        bjiv = b[j][i].v;
      } else {
        bjiu = 0.0;
        bjiv = 0.0;
      }
      x[j][i].u = lid + bjiu;
      x[j][i].v = bjiv;
    }
  }

  /* Test whether we are on the left edge of the global array */
  if (xints == 0) {
    i     = 0;
    xints = xints + 1;
    /* left edge */
    for (j=info.ys; j<info.ys+info.ym; j++) {
      if (B) {
        bjiu = b[j][i].u;
        bjiv = b[j][i].v;
      } else {
        bjiu = 0.0;
        bjiv = 0.0;
      }
      x[j][i].u = 0.0 + bjiu;
      x[j][i].v = 0.0 + bjiv;
    }
  }

  /* Test whether we are on the right edge of the global array */
  if (xinte == info.mx) {
    i     = info.mx - 1;
    xinte = xinte - 1;
    /* right edge */
    for (j=info.ys; j<info.ys+info.ym; j++) {
      if (B) {
        bjiu = b[j][i].u;
        bjiv = b[j][i].v;
      } else {
        bjiu = 0.0;
        bjiv = 0.0;
      }
      x[j][i].u = 0.0 + bjiu;
      x[j][i].v = 0.0 + bjiv;
    }
  }

  for (k=0; k < sweeps; k++) {
    for (j=info.ys; j<info.ys + info.ym; j++) {
      for (i=info.xs; i<info.xs + info.xm; i++) {
        ptconverged = PETSC_FALSE;
        pfnorm0     = 0.0;
        pfnorm      = 0.0;
        fu          = 0.0;
        fv          = 0.0;
        fomega      = 0.0;
        ftemp       = 0.0;
        for (l = 0; l < max_its && !ptconverged; l++) {
          if (B) {
            bjiu     = b[j][i].u;
            bjiv     = b[j][i].v;
            bjiomega = b[j][i].omega;
            bjitemp  = b[j][i].temp;
          } else {
            bjiu     = 0.0;
            bjiv     = 0.0;
            bjiomega = 0.0;
            bjitemp  = 0.0;
          }

          if (i != 0 && i != info.mx - 1 && j != 0 && j != info.my-1) {
            /* U velocity */
            u     = x[j][i].u;
            uxx   = (2.0*u - x[j][i-1].u - x[j][i+1].u)*hydhx;
            uyy   = (2.0*u - x[j-1][i].u - x[j+1][i].u)*hxdhy;
            fu    = uxx + uyy - .5*(x[j+1][i].omega-x[j-1][i].omega)*hx - bjiu;
            dfudu = 2.0*(hydhx + hxdhy);
            /* V velocity */
            u     = x[j][i].v;
            uxx   = (2.0*u - x[j][i-1].v - x[j][i+1].v)*hydhx;
            uyy   = (2.0*u - x[j-1][i].v - x[j+1][i].v)*hxdhy;
            fv    = uxx + uyy + .5*(x[j][i+1].omega-x[j][i-1].omega)*hy - bjiv;
            dfvdv = 2.0*(hydhx + hxdhy);
            /*
             convective coefficients for upwinding
             */
            vx  = x[j][i].u; avx = PetscAbsScalar(vx);
            vxp = .5*(vx+avx); vxm = .5*(vx-avx);
            vy  = x[j][i].v; avy = PetscAbsScalar(vy);
            vyp = .5*(vy+avy); vym = .5*(vy-avy);
            /* Omega */
            u      = x[j][i].omega;
            uxx    = (2.0*u - x[j][i-1].omega - x[j][i+1].omega)*hydhx;
            uyy    = (2.0*u - x[j-1][i].omega - x[j+1][i].omega)*hxdhy;
            fomega = uxx + uyy +  (vxp*(u - x[j][i-1].omega) + vxm*(x[j][i+1].omega - u))*hy +
                     (vyp*(u - x[j-1][i].omega) + vym*(x[j+1][i].omega - u))*hx -
                     .5*grashof*(x[j][i+1].temp - x[j][i-1].temp)*hy - bjiomega;
            /* convective coefficient derivatives */
            dfodo = 2.0*(hydhx + hxdhy) + ((vxp - vxm)*hy + (vyp - vym)*hx);
            if (PetscRealPart(vx) > 0.0) dfodu = (u - x[j][i-1].omega)*hy;
            else dfodu = (x[j][i+1].omega - u)*hy;

            if (PetscRealPart(vy) > 0.0) dfodv = (u - x[j-1][i].omega)*hx;
            else dfodv = (x[j+1][i].omega - u)*hx;

            /* Temperature */
            u     = x[j][i].temp;
            uxx   = (2.0*u - x[j][i-1].temp - x[j][i+1].temp)*hydhx;
            uyy   = (2.0*u - x[j-1][i].temp - x[j+1][i].temp)*hxdhy;
            ftemp =  uxx + uyy  + prandtl*((vxp*(u - x[j][i-1].temp) + vxm*(x[j][i+1].temp - u))*hy +
                                           (vyp*(u - x[j-1][i].temp) + vym*(x[j+1][i].temp - u))*hx) - bjitemp;
            dftdt = 2.0*(hydhx + hxdhy) + prandtl*((vxp - vxm)*hy + (vyp - vym)*hx);
            if (PetscRealPart(vx) > 0.0) dftdu = prandtl*(u - x[j][i-1].temp)*hy;
            else dftdu = prandtl*(x[j][i+1].temp - u)*hy;

            if (PetscRealPart(vy) > 0.0) dftdv = prandtl*(u - x[j-1][i].temp)*hx;
            else dftdv = prandtl*(x[j+1][i].temp - u)*hx;

            /* invert the system:
             [ dfu / du     0        0        0    ][yu] = [fu]
             [     0    dfv / dv     0        0    ][yv]   [fv]
             [ dfo / du dfo / dv dfo / do     0    ][yo]   [fo]
             [ dft / du dft / dv     0    dft / dt ][yt]   [ft]
             by simple back-substitution
           */
            yu = fu / dfudu;
            yv = fv / dfvdv;
            yo = fomega / dfodo;
            yt = ftemp / dftdt;
            yo = (fomega - (dfodu*yu + dfodv*yv)) / dfodo;
            yt = (ftemp - (dftdu*yu + dftdv*yv)) / dftdt;

            x[j][i].u     = x[j][i].u - yu;
            x[j][i].v     = x[j][i].v - yv;
            x[j][i].temp  = x[j][i].temp - yt;
            x[j][i].omega = x[j][i].omega - yo;
          }
          if (i == 0) {
            fomega        = x[j][i].omega - (x[j][i+1].v - x[j][i].v)*dhx - bjiomega;
            ftemp         = x[j][i].temp - bjitemp;
            yo            = fomega;
            yt            = ftemp;
            x[j][i].omega = x[j][i].omega - fomega;
            x[j][i].temp  = x[j][i].temp - ftemp;
          }
          if (i == info.mx - 1) {
            fomega        = x[j][i].omega - (x[j][i].v - x[j][i-1].v)*dhx - bjiomega;
            ftemp         = x[j][i].temp - (PetscReal)(grashof>0) - bjitemp;
            yo            = fomega;
            yt            = ftemp;
            x[j][i].omega = x[j][i].omega - fomega;
            x[j][i].temp  = x[j][i].temp - ftemp;
          }
          if (j == 0) {
            fomega        = x[j][i].omega + (x[j+1][i].u - x[j][i].u)*dhy - bjiomega;
            ftemp         = x[j][i].temp-x[j+1][i].temp - bjitemp;
            yo            = fomega;
            yt            = ftemp;
            x[j][i].omega = x[j][i].omega - fomega;
            x[j][i].temp  = x[j][i].temp - ftemp;
          }
          if (j == info.my - 1) {
            fomega        = x[j][i].omega + (x[j][i].u - x[j-1][i].u)*dhy - bjiomega;
            ftemp         = x[j][i].temp-x[j-1][i].temp - bjitemp;
            yo            = fomega;
            yt            = ftemp;
            x[j][i].omega = x[j][i].omega - fomega;
            x[j][i].temp  = x[j][i].temp - ftemp;
          }
          tot_its++;
          pfnorm = PetscRealPart(fu*fu + fv*fv + fomega*fomega + ftemp*ftemp);
          pfnorm = PetscSqrtReal(pfnorm);
          pynorm = PetscRealPart(yu*yu + yv*yv + yo*yo + yt*yt);
          pfnorm = PetscSqrtReal(pynorm);
          pxnorm = PetscRealPart(x[j][i].u*x[j][i].u + x[j][i].v*x[j][i].v + x[j][i].omega*x[j][i].omega + x[j][i].temp*x[j][i].temp);
          pxnorm = PetscSqrtReal(pxnorm);
          if (l == 0) pfnorm0 = pfnorm;
          if (rtol*pfnorm0 > pfnorm ||
              atol > pfnorm ||
              pxnorm*stol > pynorm) ptconverged = PETSC_TRUE;
        }
      }
    }
  }
  ierr = DMDAVecRestoreArray(da,localX,&x);CHKERRQ(ierr);
  if (B) {
    ierr = DMDAVecRestoreArray(da,localB,&b);CHKERRQ(ierr);
  }
  ierr = DMLocalToGlobalBegin(da,localX,INSERT_VALUES,X);CHKERRQ(ierr);
  ierr = DMLocalToGlobalEnd(da,localX,INSERT_VALUES,X);CHKERRQ(ierr);
  ierr = PetscLogFlops(tot_its*(84.0 + 41.0 + 26.0));CHKERRQ(ierr);
  if (B) {
    ierr = DMLocalToGlobalBegin(da,localB,INSERT_VALUES,B);CHKERRQ(ierr);
    ierr = DMLocalToGlobalEnd(da,localB,INSERT_VALUES,B);CHKERRQ(ierr);
  }
  ierr = DMRestoreLocalVector(da,&localX);CHKERRQ(ierr);
  if (B) {
    ierr = DMRestoreLocalVector(da,&localB);CHKERRQ(ierr);
  }
  PetscFunctionReturn(0);
}

From jed at jedbrown.org  Sat May 30 20:28:04 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 31 May 2015 03:28:04 +0200
Subject: [petsc-users] Modified Taylor-Hood elements with piece-wise
	constant pressure for Stokes equation
In-Reply-To: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
References: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
Message-ID: <877frpili3.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> Hello everyone,
>
> In SNES ex62, it seems to solve the Stokes equation using TH elements
> (i.e., using P2-P1 elements). However, this formulation is not locally
> conservative, and as a result many people use the P2-P1/P0 elements where
> the pressure space is enriched with a piece-wise constant to ensure mass
> conservation at the element level.

1. P2-P1disc is not stable.

2. P1disc is not "P1 enriched with piecewise constant", and the latter
does not make sense.

3. You're probably thinking of the Crouzeix-Raviart element, which has
P1disc pressure and P2 enriched with a cubic bubble for velocity.  That
element is stable and locally conservative.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/293c8e23/attachment.pgp>

From hsahasra at purdue.edu  Sat May 30 20:44:04 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Sat, 30 May 2015 21:44:04 -0400
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
Message-ID: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>

Hi,

I need to offload the linear solves in a time stepper to GPUs. I'll be
using the MAGMA sparse iterative solvers. What is the simplest way to
achieve this? Is is better to write a new KSPType or is there a KSP wrapper
which I can use?

Many thanks,
Harshad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/4dcc0138/attachment.html>

From bsmith at mcs.anl.gov  Sat May 30 21:21:41 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 30 May 2015 21:21:41 -0500
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
Message-ID: <708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>


> On May 30, 2015, at 8:44 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> 
> Hi,
> 
> I need to offload the linear solves in a time stepper to GPUs. I'll be using the MAGMA sparse iterative solvers. What is the simplest way to achieve this? Is is better to write a new KSPType or is there a KSP wrapper which I can use?

   We do not have any KSPType for the MAGMA sparse iterative solvers.

   Is your plan to use a single MPI process (that is the same as a sequential program) and ONE GPU for the solver or do you wish to have MPI parallelism for the linear solver where each MPI process uses a GPU for its part?

  Barry

> 
> Many thanks,
> Harshad


From bsmith at mcs.anl.gov  Sat May 30 21:41:04 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 30 May 2015 21:41:04 -0500
Subject: [petsc-users] NASM FormFunctionLocal
In-Reply-To: <556A5DAF.7040700@gmail.com>
References: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
	<066427A9-8A57-487A-AA9C-7583F78C7279@mcs.anl.gov>
	<556A5DAF.7040700@gmail.com>
Message-ID: <C1FC7575-641F-4879-882B-5DA6D922A5EA@mcs.anl.gov>


  Rongliang,

    It appears the NASM solver creates its own subdomains for which it calls the local function on each one:

PetscErrorCode SNESSetUp_NASM(SNES snes)
....
 if (!nasm->subsnes) {
    ierr = SNESGetDM(snes,&dm);CHKERRQ(ierr);
    if (dm) {
      nasm->usesdm = PETSC_TRUE;
      ierr         = DMCreateDomainDecomposition(dm,&nasm->n,NULL,NULL,NULL,&subdms);CHKERRQ(ierr);
      if (!subdms) SETERRQ(PetscObjectComm((PetscObject)dm),PETSC_ERR_ARG_WRONGSTATE,"DM has no default decomposition defined.  Set subsolves manually with SNESNASMSetSubdomains().");
      ierr = DMCreateDomainDecompositionScatters(dm,nasm->n,subdms,&nasm->iscatter,&nasm->oscatter,&nasm->gscatter);CHKERRQ(ierr);

      ierr = SNESGetOptionsPrefix(snes, &optionsprefix);CHKERRQ(ierr);
      ierr = PetscMalloc1(nasm->n,&nasm->subsnes);CHKERRQ(ierr);
      for (i=0; i<nasm->n; i++) {
        ierr = SNESCreate(PETSC_COMM_SELF,&nasm->subsnes[i]);CHKERRQ(ierr);

these subdomains do not correspond to the "normal" one subdomain per process decomposition of the domain that you expect when you call 

ierr = DMDAVecGetArray(info->da,user->temp, &z);CHKERRQ(ierr); // added by rlchen

inside PetscErrorCode FormFunctionLocal(DMDALocalInfo *info,Field **x,Field **f,void *ptr)

In other words you are expecting that the info->da passed into FormFunctionLocal() is the same as the DA in your original main() program. But with NASM solver it simply is not, it is some other DA that is created by the NASM solver. 

If you want to pass some vector like your .temp vector into your local function you will need to understand exactly how the NASM solver works so you can pass the correct vectors in. Simple passing a vector associated with the original DA of the main program won't work.

  Barry




> On May 30, 2015, at 8:02 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
> 
> Hi Barry and Matt,
> 
> Many thanks for your reply. I only have one DA and I do not think I changed others when I changed the overlap.
> 
> This error can be reproduced using the attached code (added some lines in the src/snes/examples/tutorials/ex19.c) run with:
> 
> runex19_1:
>        -@${MPIEXEC} -n 2 ./ex19 -da_refine 3 -snes_type nasm -da_overlap 2 -snes_monitor
> 
> The full error messages are followed:
> ------------------
> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
>  0 SNES Function norm 4.066115181565e-02
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Arguments are incompatible
> [1]PETSC ERROR: Vector local size 1300 is not compatible with DMDA local sizes 1400 1728
> 
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown
> [1]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 31 08:55:16 2015
> [1]PETSC ERROR: Configure options --download-fblaslapack --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 --with-debugging=1 --with-shared-libraries=0
> [1]PETSC ERROR: #1 DMDAVecGetArray() line 71 in /home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
> [1]PETSC ERROR: #2 FormFunctionLocal() line 275 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
> [1]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in /home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
> [1]PETSC ERROR: #4 SNESComputeFunction() line 2033 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [1]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #6 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [1]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
> [1]PETSC ERROR: #8 SNESSolve_NASM() line 859 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
> [1]PETSC ERROR: #9 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [1]PETSC ERROR: #10 main() line 162 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 75) - process 1
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: Vector local size 1400 is not compatible with DMDA local sizes 1500 1836
> 
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown
> [0]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 31 08:55:16 2015
> [0]PETSC ERROR: Configure options --download-fblaslapack --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 --with-debugging=1 --with-shared-libraries=0
> [0]PETSC ERROR: #1 DMDAVecGetArray() line 71 in /home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
> [0]PETSC ERROR: #2 FormFunctionLocal() line 275 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
> [0]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in /home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
> [0]PETSC ERROR: #4 SNESComputeFunction() line 2033 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [0]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
> [0]PETSC ERROR: #8 SNESSolve_NASM() line 859 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
> [0]PETSC ERROR: #9 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
> [0]PETSC ERROR: #10 main() line 162 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 75) - process 0
> 
> =====================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   EXIT CODE: 19200
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> =====================================================================================
> make: [runex19_1] Error 75 (ignored)
> -----------------------
> 
> Best regards,
> Rongliang
> 
> On 05/31/2015 01:09 AM, Barry Smith wrote:
>>> On May 30, 2015, at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>> 
>>> Hi there,
>>> 
>>> I am trying to use NASM to solve a nonlinear problem. In my FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the "-da_overlap 0" case, the code works fine.
>>> 
>>> For the "-da_overlap 1" case, when I use the vector "myvec" created by DMCreateLocalVector, it works fine. But if I use the vector "myvec" created by DMCreateGlobalVector, I got the error messages:
>>> -----------------------
>>> [6]Arguments are incompatible
>>> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local sizes 619528 664392
>> I think you are changing something else at the same time you changed the overlap.  The number ^^^^  619528 is the local size of the global vector and it should remain the same when you change the overlap but below you have TWO other numbers 633144 and 650256  Do you have more than one DMDA?
>> Run changing absolutely nothing except the overlap.
>>> ----------------------
>>> 
>>> And when I switch to "-da_overlap 2", both of the global and local version of "mrvec" got error messages:
>>> ----------------------------------
>>> [10]PETSC ERROR: Arguments are incompatible
>>> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local sizes 633144 678680
>>> 
>>> [3]PETSC ERROR: Arguments are incompatible
>>> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local sizes 650256 696540
>>> -----------------------------------
>>> 
>>> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for NASM? Thanks.
>>> 
>>> Best regards,
>>> Rongliang
>>> 
> 
> <ex19.c>


From hsahasra at purdue.edu  Sat May 30 21:55:14 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Sat, 30 May 2015 22:55:14 -0400
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
Message-ID: <CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>

>
>  Is your plan to use a single MPI process (that is the same as a
> sequential program) and ONE GPU for the solver or do you wish to have MPI
> parallelism for the linear solver where each MPI process uses a GPU for its
> part?


For now, I want to serialize the matrices and vectors and offload them to 1
GPU from the root process. Then distribute the result later.

Harshad

On Sat, May 30, 2015 at 10:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On May 30, 2015, at 8:44 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu>
> wrote:
> >
> > Hi,
> >
> > I need to offload the linear solves in a time stepper to GPUs. I'll be
> using the MAGMA sparse iterative solvers. What is the simplest way to
> achieve this? Is is better to write a new KSPType or is there a KSP wrapper
> which I can use?
>
>    We do not have any KSPType for the MAGMA sparse iterative solvers.
>
>    Is your plan to use a single MPI process (that is the same as a
> sequential program) and ONE GPU for the solver or do you wish to have MPI
> parallelism for the linear solver where each MPI process uses a GPU for its
> part?
>
>   Barry
>
> >
> > Many thanks,
> > Harshad
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/097fe93e/attachment.html>

From jed at jedbrown.org  Sat May 30 22:01:02 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 31 May 2015 05:01:02 +0200
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
Message-ID: <871thxih75.fsf@jedbrown.org>

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
> For now, I want to serialize the matrices and vectors and offload them to 1
> GPU from the root process. Then distribute the result later.

Unless you have experience with these solvers and the overheads
involved, I think you should expect this to be much slower than simply
doing the solves using a reasonable method in the CPU.  Is your intent
to solve a problem that matters in a way that makes sense for a
scientist or engineer, or is it to demonstrate that a particular
combination of packages/methods/hardware can be used?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/c92291fe/attachment.pgp>

From hsahasra at purdue.edu  Sat May 30 22:14:19 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Sat, 30 May 2015 23:14:19 -0400
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <871thxih75.fsf@jedbrown.org>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
Message-ID: <CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>

>
> Is your intent to solve a problem that matters in a way that makes sense
> for a scientist or engineer


I want to see if we can speed up the time stepper for a large system using
GPUs. For large systems with sparse matrix of size 420,000^2, each time
step takes 341 sec on a single process and 180 seconds on 16 processes. So
the scaling isn't that good. We also run out of memory with more number of
processes.

On Sat, May 30, 2015 at 11:01 PM, Jed Brown <jed at jedbrown.org> wrote:

> Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
> > For now, I want to serialize the matrices and vectors and offload them
> to 1
> > GPU from the root process. Then distribute the result later.
>
> Unless you have experience with these solvers and the overheads
> involved, I think you should expect this to be much slower than simply
> doing the solves using a reasonable method in the CPU.  Is your intent
> to solve a problem that matters in a way that makes sense for a
> scientist or engineer, or is it to demonstrate that a particular
> combination of packages/methods/hardware can be used?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/81782a72/attachment-0001.html>

From jychang48 at gmail.com  Sat May 30 22:21:01 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Sat, 30 May 2015 22:21:01 -0500
Subject: [petsc-users] Modified Taylor-Hood elements with piece-wise
 constant pressure for Stokes equation
In-Reply-To: <877frpili3.fsf@jedbrown.org>
References: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
	<877frpili3.fsf@jedbrown.org>
Message-ID: <CAP2=TMgqO+UZc_C8_CrA4A+8DJxXm-vDzCGEGaxuwOVFq4uWmA@mail.gmail.com>

I am referring to P2 / (P1 + P0) elements, I think this is the correct way
of expressing it. Some call it modified Taylor Hood, others call it
something else, but it's not Crouzeix-Raviart elements.

On Sat, May 30, 2015 at 8:28 PM, Jed Brown <jed at jedbrown.org> wrote:

> Justin Chang <jychang48 at gmail.com> writes:
>
> > Hello everyone,
> >
> > In SNES ex62, it seems to solve the Stokes equation using TH elements
> > (i.e., using P2-P1 elements). However, this formulation is not locally
> > conservative, and as a result many people use the P2-P1/P0 elements where
> > the pressure space is enriched with a piece-wise constant to ensure mass
> > conservation at the element level.
>
> 1. P2-P1disc is not stable.
>
> 2. P1disc is not "P1 enriched with piecewise constant", and the latter
> does not make sense.
>
> 3. You're probably thinking of the Crouzeix-Raviart element, which has
> P1disc pressure and P2 enriched with a cubic bubble for velocity.  That
> element is stable and locally conservative.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/fc06e6df/attachment.html>

From jed at jedbrown.org  Sat May 30 22:22:40 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 31 May 2015 05:22:40 +0200
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
	<CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
Message-ID: <87y4k5h1mn.fsf@jedbrown.org>

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:

>>
>> Is your intent to solve a problem that matters in a way that makes sense
>> for a scientist or engineer
>
>
> I want to see if we can speed up the time stepper for a large system using
> GPUs. For large systems with sparse matrix of size 420,000^2, each time
> step takes 341 sec on a single process and 180 seconds on 16 processes. So
> the scaling isn't that good. 

 Surely you're familiar with this.

http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

Is the poor scaling due to increased iteration count?  What method are
you using?

> We also run out of memory with more number of processes.

This sounds like a problem with your code (non-scalable data structure).

Also, the GPU doesn't have more memory than the CPU.

How long does it take to solve that system stand-alone using MAGMA,
including the data transfers?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/cae5fb7c/attachment.pgp>

From hsahasra at purdue.edu  Sat May 30 22:33:02 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Sat, 30 May 2015 23:33:02 -0400
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <87y4k5h1mn.fsf@jedbrown.org>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
	<CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
	<87y4k5h1mn.fsf@jedbrown.org>
Message-ID: <CAC0ZLSq6o9YTGLtoQ2JVDHfFdJVrgJzGBv2bSrnNozSMRHSm5A@mail.gmail.com>

>
>  Surely you're familiar with this.


Yes, I'm familiar with this. We are running on Intel Xeon E5 processor. It
has enough bandwidth and performance. Also, we are just running on one node
currently.

Is the poor scaling due to increased iteration count?  What method are you
> using?


This is exactly why we have poor scaling. We have tried KSPGMRES.

This sounds like a problem with your code (non-scalable data structure).


We need to work on the algorithm for matrix assembly. In it's current
state, one CPU ends up doing much of the work.This could be the cause of
bad memory scaling. This doesn't contribute to the bad scaling to time
stepping, time taken for time stepping is counted separately from assembly.

How long does it take to solve that system stand-alone using MAGMA, including
> the data transfers?


I'm still working on these tests.

On Sat, May 30, 2015 at 11:22 PM, Jed Brown <jed at jedbrown.org> wrote:

> Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
>
> >>
> >> Is your intent to solve a problem that matters in a way that makes sense
> >> for a scientist or engineer
> >
> >
> > I want to see if we can speed up the time stepper for a large system
> using
> > GPUs. For large systems with sparse matrix of size 420,000^2, each time
> > step takes 341 sec on a single process and 180 seconds on 16 processes.
> So
> > the scaling isn't that good.
>
>  Surely you're familiar with this.
>
> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>
> Is the poor scaling due to increased iteration count?  What method are
> you using?
>
> > We also run out of memory with more number of processes.
>
> This sounds like a problem with your code (non-scalable data structure).
>
> Also, the GPU doesn't have more memory than the CPU.
>
> How long does it take to solve that system stand-alone using MAGMA,
> including the data transfers?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150530/d75644c8/attachment.html>

From jed at jedbrown.org  Sat May 30 22:50:01 2015
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 31 May 2015 05:50:01 +0200
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <CAC0ZLSq6o9YTGLtoQ2JVDHfFdJVrgJzGBv2bSrnNozSMRHSm5A@mail.gmail.com>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
	<CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
	<87y4k5h1mn.fsf@jedbrown.org>
	<CAC0ZLSq6o9YTGLtoQ2JVDHfFdJVrgJzGBv2bSrnNozSMRHSm5A@mail.gmail.com>
Message-ID: <87vbf9h0d2.fsf@jedbrown.org>

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:

>>
>>  Surely you're familiar with this.
>
>
> Yes, I'm familiar with this. We are running on Intel Xeon E5 processor. It
> has enough bandwidth and performance. 

One core saturates a sizeable fraction of the memory bandwidth for the
socket.  You certainly can't expect 10x speedups when moving from 1 to
16 cores for a memory bandwidth limited application.

> Is the poor scaling due to increased iteration count?  What method are you
>> using?
>
> This is exactly why we have poor scaling. We have tried KSPGMRES.

GMRES is secondary for this discussion; which preconditioner are you
using and how many iterations does it require?

> This sounds like a problem with your code (non-scalable data structure).
>
> We need to work on the algorithm for matrix assembly. In it's current
> state, one CPU ends up doing much of the work.This could be the cause of
> bad memory scaling. This doesn't contribute to the bad scaling to time
> stepping, time taken for time stepping is counted separately from assembly.

This is a linear autonomous system?

> How long does it take to solve that system stand-alone using MAGMA, including
>> the data transfers?
>
> I'm still working on these tests.

Do that first.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/9a3bcb0c/attachment.pgp>

From hsahasra at purdue.edu  Sat May 30 23:02:40 2015
From: hsahasra at purdue.edu (Harshad Sahasrabudhe)
Date: Sun, 31 May 2015 00:02:40 -0400
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <87vbf9h0d2.fsf@jedbrown.org>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
	<CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
	<87y4k5h1mn.fsf@jedbrown.org>
	<CAC0ZLSq6o9YTGLtoQ2JVDHfFdJVrgJzGBv2bSrnNozSMRHSm5A@mail.gmail.com>
	<87vbf9h0d2.fsf@jedbrown.org>
Message-ID: <CAC0ZLSqEnWCNeUZ3w3qBd5xKMPpJX9Zm-CA9za3_f8cwVcgrUg@mail.gmail.com>

>
> which preconditioner are you using and how many iterations does it
> require?

 This is a linear autonomous system?



> How long does it take to solve that system stand-alone using MAGMA,
> including
> >> the data transfers?
> >
> > I'm still working on these tests.
> Do that first.


Thank you very much for the guidance. I'll get back with the answers
tomorrow.


On Sat, May 30, 2015 at 11:50 PM, Jed Brown <jed at jedbrown.org> wrote:

> Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
>
> >>
> >>  Surely you're familiar with this.
> >
> >
> > Yes, I'm familiar with this. We are running on Intel Xeon E5 processor.
> It
> > has enough bandwidth and performance.
>
> One core saturates a sizeable fraction of the memory bandwidth for the
> socket.  You certainly can't expect 10x speedups when moving from 1 to
> 16 cores for a memory bandwidth limited application.
>
> > Is the poor scaling due to increased iteration count?  What method are
> you
> >> using?
> >
> > This is exactly why we have poor scaling. We have tried KSPGMRES.
>
> GMRES is secondary for this discussion; which preconditioner are you
> using and how many iterations does it require?
>
> > This sounds like a problem with your code (non-scalable data structure).
> >
> > We need to work on the algorithm for matrix assembly. In it's current
> > state, one CPU ends up doing much of the work.This could be the cause of
> > bad memory scaling. This doesn't contribute to the bad scaling to time
> > stepping, time taken for time stepping is counted separately from
> assembly.
>
> This is a linear autonomous system?
>
> > How long does it take to solve that system stand-alone using MAGMA,
> including
> >> the data transfers?
> >
> > I'm still working on these tests.
>
> Do that first.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/5c01f67e/attachment.html>

From venkateshgk.j at gmail.com  Sun May 31 07:37:20 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sun, 31 May 2015 18:07:20 +0530
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CADZ9mvkihmmyMvjVDW5GV7FsFj0XPwy=o6maV+SbM4KuKZKEvQ@mail.gmail.com>
	<CABzT7hNfgcO_hknFxmVpuZx5GZUbp0mEmR6Dmmjqp7myWHOqBw@mail.gmail.com>
	<CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
	<CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>
Message-ID: <CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>

Hi all,

I tried to run my Generalized Eigenproblem in 120 x 24 = 2880 cores.
The matrix size of A = 20GB and B = 5GB.

It got killed after 7 Hrs of run time. Please see the mumps error log. Why
must it fail ?
I gave the command:

aprun -n 240 -N 24 ./ex7 -f1 a110t -f2 b110t -st_type sinvert -eps_nev 1
-log_summary -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_package mumps -mat_mumps_cntl_1 1e-2

Kindly let me know.

cheers,
Venkatesh

On Fri, May 29, 2015 at 10:46 PM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi Matt, users,
>
> Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get
> into the segmentation error if I increase my matrix size.
>
> Can you suggest other software for direct solver for QR in parallel since
> as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am
> attaching the working version mumps log.
>
> My matrix size here is around 47000x47000. If I am not wrong, the memory
> usage per core is 272MB.
>
> Can you tell me if I am wrong ? or really if its light on memory for this
> matrix ?
>
> Thanks
> cheers,
> Venkatesh
>
> On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <matt.landreman at gmail.com>
> wrote:
>
>> Dear Venkatesh,
>>
>> As you can see in the error log, you are now getting a segmentation
>> fault, which is almost certainly a separate issue from the info(1)=-9
>> memory problem you had previously. Here is one idea which may or may not
>> help. I've used mumps on the NERSC Edison system, and I found that I
>> sometimes get segmentation faults when using the default Intel compiler.
>> When I switched to the cray compiler the problem disappeared. So you could
>> perhaps try a different compiler if one is available on your system.
>>
>> Matt
>> On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j at gmail.com> wrote:
>>
>>> Hi Matt,
>>>
>>> I did what you told and read the manual of that CNTL parameters. I solve
>>> for that with CNTL(1)=1e-4. It is working.
>>>
>>> But it was a test matrix with size 46000x46000. Actual matrix size is
>>> 108900x108900 and will increase in the future.
>>>
>>> I get this error of memory allocation failed. And the binary matrix size
>>> of A is 20GB and B is 5 GB.
>>>
>>> Now I submit this in 240 processors each 4 GB RAM and also in 128
>>> Processors with total 512 GB RAM.
>>>
>>> In both the cases, it fails with the following error like memory is not
>>> enough. But for 90000x90000 size it had run serially in Matlab with <256 GB
>>> RAM.
>>>
>>> Kindly let me know.
>>>
>>> Venkatesh
>>>
>>> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <
>>> matt.landreman at gmail.com> wrote:
>>>
>>>> Hi Venkatesh,
>>>>
>>>> I've struggled a bit with mumps memory allocation too.  I think the
>>>> behavior of mumps is roughly the following. First, in the "analysis step",
>>>> mumps computes a minimum memory required based on the structure of nonzeros
>>>> in the matrix.  Then when it actually goes to factorize the matrix, if it
>>>> ever encounters an element smaller than CNTL(1) (default=0.01) in the
>>>> diagonal of a sub-matrix it is trying to factorize, it modifies the
>>>> ordering to avoid the small pivot, which increases the fill-in (hence
>>>> memory needed).  ICNTL(14) sets the margin allowed for this unanticipated
>>>> fill-in.  Setting ICNTL(14)=200000 as in your email is not the solution,
>>>> since this means mumps asks for a huge amount of memory at the start.
>>>> Better would be to lower CNTL(1) or (I think) use static pivoting
>>>> (CNTL(4)).  Read the section in the mumps manual about these CNTL
>>>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the
>>>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14).
>>>>
>>>> Also, I recommend running with ICNTL(4)=3 to display diagnostics. Look
>>>> for the line in standard output that says "TOTAL     space in MBYTES for IC
>>>> factorization".  This is the amount of memory that mumps is trying to
>>>> allocate, and for the default ICNTL(14), it should be similar to matlab's
>>>> need.
>>>>
>>>> Hope this helps,
>>>> -Matt Landreman
>>>> University of Maryland
>>>>
>>>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>>
>>>>> I am solving a large generalized Eigenvalue problem.
>>>>>
>>>>> I am getting the following error which is attached, after giving the
>>>>> command:
>>>>>
>>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>> 200000
>>>>>
>>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>>> asking like around 70 GB per processor.
>>>>>
>>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>>
>>>>> After trying out superLU_dist, I have attached the error there also
>>>>> (segmentation error).
>>>>>
>>>>> Kindly help me.
>>>>>
>>>>> Venkatesh
>>>>>
>>>>>
>>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/e4dbb44a/attachment-0001.html>
-------------- next part --------------

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
Entering ZMUMPS driver with JOB, N, NZ =   1      108900              0

 ZMUMPS 4.10.0        
L U Solver for unsymmetric matrices
Type of parallelism: Working host

 ****** ANALYSIS STEP ********

 ** Max-trans not allowed because matrix is distributed
 ... Structural symmetry (in percent)=   70
 Density: NBdense, Average, Median   =    01232012102
 Ordering based on METIS 
 A root of estimated size        41878  has been selected for Scalapack.

Leaving analysis phase with  ...
INFOG(1)                                       =               0
INFOG(2)                                       =               0
 -- (20) Number of entries in factors (estim.) =      5475631310
 --  (3) Storage of factors  (REAL, estimated) =     89125045501
 --  (4) Storage of factors  (INT , estimated) =       655485547
 --  (5) Maximum frontal size      (estimated) =           41878
 --  (6) Number of nodes in the tree           =             471
 -- (32) Type of analysis effectively used     =               1
 --  (7) Ordering option effectively used      =               5
ICNTL(6) Maximum transversal option            =               0
ICNTL(7) Pivot order option                    =               7
Percentage of memory relaxation (effective)    =              35
Number of level 2 nodes                        =             439
Number of split nodes                          =             149
RINFOG(1) Operations during elimination (estim)=   1.584D+14
Distributed matrix entry format (ICNTL(18))    =               3
 ** Rank of proc needing largest memory in IC facto        :         0
 ** Estimated corresponding MBYTES for IC facto            :     33885
 ** Estimated avg. MBYTES per work. proc at facto (IC)     :      8679
 ** TOTAL     space in MBYTES for IC factorization         :  24997648
 ** Rank of proc needing largest memory for OOC facto      :         0
 ** Estimated corresponding MBYTES for OOC facto           :     33683
 ** Estimated avg. MBYTES per work. proc at facto (OOC)    :      8398
 ** TOTAL     space in MBYTES for OOC factorization        :  24187332
Entering ZMUMPS driver with JOB, N, NZ =   2      108900     1035808400

 ****** FACTORIZATION STEP ********


 GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...
 NUMBER OF WORKING PROCESSES              =        2880
 OUT-OF-CORE OPTION (ICNTL(22))           =           0
 REAL SPACE FOR FACTORS                   = 89125045501
 INTEGER SPACE FOR FACTORS                =   655485547
 MAXIMUM FRONTAL SIZE (ESTIMATED)         =       41878
 NUMBER OF NODES IN THE TREE              =         471
 Convergence error after scaling for ONE-NORM (option 7/8)   = 0.95D+00
 Maximum effective relaxed size of S              =  1630229452
 Average effective relaxed size of S              =   225206115
[NID 01214] 2015-05-31 17:36:18 Apid 409924: initiated application termination
[NID 01214] 2015-05-31 17:34:59 Apid 409924: OOM killer terminated this process.
Application 409924 exit signals: Killed
Application 409924 resources: utime ~0s, stime ~225s, Rss ~7716, inblocks ~192480, outblocks ~28560

From venkateshgk.j at gmail.com  Sun May 31 07:38:18 2015
From: venkateshgk.j at gmail.com (venkatesh g)
Date: Sun, 31 May 2015 18:08:18 +0530
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CADZ9mvkihmmyMvjVDW5GV7FsFj0XPwy=o6maV+SbM4KuKZKEvQ@mail.gmail.com>
	<CABzT7hNfgcO_hknFxmVpuZx5GZUbp0mEmR6Dmmjqp7myWHOqBw@mail.gmail.com>
	<CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
	<CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>
	<CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>
Message-ID: <CABzT7hM=tznfuoPbbZevxtVerme30jR6OrsBiJ8i-DEm66XsDw@mail.gmail.com>

Sorry the command was:
aprun -n 2880 -N 24 ./ex7 -f1 a110t -f2 b110t -st_type sinvert -eps_nev 1
-st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps
-mat_mumps_cntl_1 1e-6 -mat_mumps_icntl_4 2

cheers,
Venkatesh

On Sun, May 31, 2015 at 6:07 PM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi all,
>
> I tried to run my Generalized Eigenproblem in 120 x 24 = 2880 cores.
> The matrix size of A = 20GB and B = 5GB.
>
> It got killed after 7 Hrs of run time. Please see the mumps error log. Why
> must it fail ?
> I gave the command:
>
> aprun -n 240 -N 24 ./ex7 -f1 a110t -f2 b110t -st_type sinvert -eps_nev 1
> -log_summary -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps -mat_mumps_cntl_1 1e-2
>
> Kindly let me know.
>
> cheers,
> Venkatesh
>
> On Fri, May 29, 2015 at 10:46 PM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi Matt, users,
>>
>> Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get
>> into the segmentation error if I increase my matrix size.
>>
>> Can you suggest other software for direct solver for QR in parallel since
>> as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am
>> attaching the working version mumps log.
>>
>> My matrix size here is around 47000x47000. If I am not wrong, the memory
>> usage per core is 272MB.
>>
>> Can you tell me if I am wrong ? or really if its light on memory for this
>> matrix ?
>>
>> Thanks
>> cheers,
>> Venkatesh
>>
>> On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <matt.landreman at gmail.com
>> > wrote:
>>
>>> Dear Venkatesh,
>>>
>>> As you can see in the error log, you are now getting a segmentation
>>> fault, which is almost certainly a separate issue from the info(1)=-9
>>> memory problem you had previously. Here is one idea which may or may not
>>> help. I've used mumps on the NERSC Edison system, and I found that I
>>> sometimes get segmentation faults when using the default Intel compiler.
>>> When I switched to the cray compiler the problem disappeared. So you could
>>> perhaps try a different compiler if one is available on your system.
>>>
>>> Matt
>>> On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j at gmail.com> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> I did what you told and read the manual of that CNTL parameters. I
>>>> solve for that with CNTL(1)=1e-4. It is working.
>>>>
>>>> But it was a test matrix with size 46000x46000. Actual matrix size is
>>>> 108900x108900 and will increase in the future.
>>>>
>>>> I get this error of memory allocation failed. And the binary matrix
>>>> size of A is 20GB and B is 5 GB.
>>>>
>>>> Now I submit this in 240 processors each 4 GB RAM and also in 128
>>>> Processors with total 512 GB RAM.
>>>>
>>>> In both the cases, it fails with the following error like memory is not
>>>> enough. But for 90000x90000 size it had run serially in Matlab with <256 GB
>>>> RAM.
>>>>
>>>> Kindly let me know.
>>>>
>>>> Venkatesh
>>>>
>>>> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <
>>>> matt.landreman at gmail.com> wrote:
>>>>
>>>>> Hi Venkatesh,
>>>>>
>>>>> I've struggled a bit with mumps memory allocation too.  I think the
>>>>> behavior of mumps is roughly the following. First, in the "analysis step",
>>>>> mumps computes a minimum memory required based on the structure of nonzeros
>>>>> in the matrix.  Then when it actually goes to factorize the matrix, if it
>>>>> ever encounters an element smaller than CNTL(1) (default=0.01) in the
>>>>> diagonal of a sub-matrix it is trying to factorize, it modifies the
>>>>> ordering to avoid the small pivot, which increases the fill-in (hence
>>>>> memory needed).  ICNTL(14) sets the margin allowed for this unanticipated
>>>>> fill-in.  Setting ICNTL(14)=200000 as in your email is not the solution,
>>>>> since this means mumps asks for a huge amount of memory at the start.
>>>>> Better would be to lower CNTL(1) or (I think) use static pivoting
>>>>> (CNTL(4)).  Read the section in the mumps manual about these CNTL
>>>>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the
>>>>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14).
>>>>>
>>>>> Also, I recommend running with ICNTL(4)=3 to display diagnostics. Look
>>>>> for the line in standard output that says "TOTAL     space in MBYTES for IC
>>>>> factorization".  This is the amount of memory that mumps is trying to
>>>>> allocate, and for the default ICNTL(14), it should be similar to matlab's
>>>>> need.
>>>>>
>>>>> Hope this helps,
>>>>> -Matt Landreman
>>>>> University of Maryland
>>>>>
>>>>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>>>
>>>>>> I am solving a large generalized Eigenvalue problem.
>>>>>>
>>>>>> I am getting the following error which is attached, after giving the
>>>>>> command:
>>>>>>
>>>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>> 200000
>>>>>>
>>>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>>>> asking like around 70 GB per processor.
>>>>>>
>>>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>>>
>>>>>> After trying out superLU_dist, I have attached the error there also
>>>>>> (segmentation error).
>>>>>>
>>>>>> Kindly help me.
>>>>>>
>>>>>> Venkatesh
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/a7b2e658/attachment.html>

From dave.mayhem23 at gmail.com  Sun May 31 08:36:14 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sun, 31 May 2015 15:36:14 +0200
Subject: [petsc-users]  MUMPS error and superLU error
In-Reply-To: <CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CADZ9mvkihmmyMvjVDW5GV7FsFj0XPwy=o6maV+SbM4KuKZKEvQ@mail.gmail.com>
	<CABzT7hNfgcO_hknFxmVpuZx5GZUbp0mEmR6Dmmjqp7myWHOqBw@mail.gmail.com>
	<CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
	<CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>
	<CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>
Message-ID: <CAJ98EDoM13ezx41s+DKFik+3Xod5_v=duNJMS6BXgn+x=e_0ew@mail.gmail.com>

It failed due to a lack of memory. "OOM" stands for "out of memory". OOM
killer terminated your job means you ran out of memory.




On Sunday, 31 May 2015, venkatesh g <venkateshgk.j at gmail.com
<javascript:_e(%7B%7D,'cvml','venkateshgk.j at gmail.com');>> wrote:

> Hi all,
>
> I tried to run my Generalized Eigenproblem in 120 x 24 = 2880 cores.
> The matrix size of A = 20GB and B = 5GB.
>
> It got killed after 7 Hrs of run time. Please see the mumps error log. Why
> must it fail ?
> I gave the command:
>
> aprun -n 240 -N 24 ./ex7 -f1 a110t -f2 b110t -st_type sinvert -eps_nev 1
> -log_summary -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_package mumps -mat_mumps_cntl_1 1e-2
>
> Kindly let me know.
>
> cheers,
> Venkatesh
>
> On Fri, May 29, 2015 at 10:46 PM, venkatesh g <venkateshgk.j at gmail.com>
> wrote:
>
>> Hi Matt, users,
>>
>> Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get
>> into the segmentation error if I increase my matrix size.
>>
>> Can you suggest other software for direct solver for QR in parallel since
>> as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am
>> attaching the working version mumps log.
>>
>> My matrix size here is around 47000x47000. If I am not wrong, the memory
>> usage per core is 272MB.
>>
>> Can you tell me if I am wrong ? or really if its light on memory for this
>> matrix ?
>>
>> Thanks
>> cheers,
>> Venkatesh
>>
>> On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <matt.landreman at gmail.com
>> > wrote:
>>
>>> Dear Venkatesh,
>>>
>>> As you can see in the error log, you are now getting a segmentation
>>> fault, which is almost certainly a separate issue from the info(1)=-9
>>> memory problem you had previously. Here is one idea which may or may not
>>> help. I've used mumps on the NERSC Edison system, and I found that I
>>> sometimes get segmentation faults when using the default Intel compiler.
>>> When I switched to the cray compiler the problem disappeared. So you could
>>> perhaps try a different compiler if one is available on your system.
>>>
>>> Matt
>>> On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j at gmail.com> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> I did what you told and read the manual of that CNTL parameters. I
>>>> solve for that with CNTL(1)=1e-4. It is working.
>>>>
>>>> But it was a test matrix with size 46000x46000. Actual matrix size is
>>>> 108900x108900 and will increase in the future.
>>>>
>>>> I get this error of memory allocation failed. And the binary matrix
>>>> size of A is 20GB and B is 5 GB.
>>>>
>>>> Now I submit this in 240 processors each 4 GB RAM and also in 128
>>>> Processors with total 512 GB RAM.
>>>>
>>>> In both the cases, it fails with the following error like memory is not
>>>> enough. But for 90000x90000 size it had run serially in Matlab with <256 GB
>>>> RAM.
>>>>
>>>> Kindly let me know.
>>>>
>>>> Venkatesh
>>>>
>>>> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <
>>>> matt.landreman at gmail.com> wrote:
>>>>
>>>>> Hi Venkatesh,
>>>>>
>>>>> I've struggled a bit with mumps memory allocation too.  I think the
>>>>> behavior of mumps is roughly the following. First, in the "analysis step",
>>>>> mumps computes a minimum memory required based on the structure of nonzeros
>>>>> in the matrix.  Then when it actually goes to factorize the matrix, if it
>>>>> ever encounters an element smaller than CNTL(1) (default=0.01) in the
>>>>> diagonal of a sub-matrix it is trying to factorize, it modifies the
>>>>> ordering to avoid the small pivot, which increases the fill-in (hence
>>>>> memory needed).  ICNTL(14) sets the margin allowed for this unanticipated
>>>>> fill-in.  Setting ICNTL(14)=200000 as in your email is not the solution,
>>>>> since this means mumps asks for a huge amount of memory at the start.
>>>>> Better would be to lower CNTL(1) or (I think) use static pivoting
>>>>> (CNTL(4)).  Read the section in the mumps manual about these CNTL
>>>>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the
>>>>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14).
>>>>>
>>>>> Also, I recommend running with ICNTL(4)=3 to display diagnostics. Look
>>>>> for the line in standard output that says "TOTAL     space in MBYTES for IC
>>>>> factorization".  This is the amount of memory that mumps is trying to
>>>>> allocate, and for the default ICNTL(14), it should be similar to matlab's
>>>>> need.
>>>>>
>>>>> Hope this helps,
>>>>> -Matt Landreman
>>>>> University of Maryland
>>>>>
>>>>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>>>
>>>>>> I am solving a large generalized Eigenvalue problem.
>>>>>>
>>>>>> I am getting the following error which is attached, after giving the
>>>>>> command:
>>>>>>
>>>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>> 200000
>>>>>>
>>>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>>>> asking like around 70 GB per processor.
>>>>>>
>>>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>>>
>>>>>> After trying out superLU_dist, I have attached the error there also
>>>>>> (segmentation error).
>>>>>>
>>>>>> Kindly help me.
>>>>>>
>>>>>> Venkatesh
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/29f55383/attachment.html>

From rongliang.chan at gmail.com  Sun May 31 09:05:11 2015
From: rongliang.chan at gmail.com (Rongliang Chen)
Date: Sun, 31 May 2015 22:05:11 +0800
Subject: [petsc-users] NASM FormFunctionLocal
In-Reply-To: <C1FC7575-641F-4879-882B-5DA6D922A5EA@mcs.anl.gov>
References: <CAOoA1QUTVYwBxr6c2NkztK+OZqrkGQsj9hP8_6E4XyPKWxebAg@mail.gmail.com>
	<066427A9-8A57-487A-AA9C-7583F78C7279@mcs.anl.gov>
	<556A5DAF.7040700@gmail.com>
	<C1FC7575-641F-4879-882B-5DA6D922A5EA@mcs.anl.gov>
Message-ID: <556B1517.20802@gmail.com>

Hi Barry,

Thanks for your reply.

I have no idea how to pass a vector into the FormFunctionLocal() right 
now. I will try to read the NASM source code and hope can find a way to 
do it.

Best regards,
Rongliang

On 05/31/2015 10:41 AM, Barry Smith wrote:
>    Rongliang,
>
>      It appears the NASM solver creates its own subdomains for which it calls the local function on each one:
>
> PetscErrorCode SNESSetUp_NASM(SNES snes)
> ....
>   if (!nasm->subsnes) {
>      ierr = SNESGetDM(snes,&dm);CHKERRQ(ierr);
>      if (dm) {
>        nasm->usesdm = PETSC_TRUE;
>        ierr         = DMCreateDomainDecomposition(dm,&nasm->n,NULL,NULL,NULL,&subdms);CHKERRQ(ierr);
>        if (!subdms) SETERRQ(PetscObjectComm((PetscObject)dm),PETSC_ERR_ARG_WRONGSTATE,"DM has no default decomposition defined.  Set subsolves manually with SNESNASMSetSubdomains().");
>        ierr = DMCreateDomainDecompositionScatters(dm,nasm->n,subdms,&nasm->iscatter,&nasm->oscatter,&nasm->gscatter);CHKERRQ(ierr);
>
>        ierr = SNESGetOptionsPrefix(snes, &optionsprefix);CHKERRQ(ierr);
>        ierr = PetscMalloc1(nasm->n,&nasm->subsnes);CHKERRQ(ierr);
>        for (i=0; i<nasm->n; i++) {
>          ierr = SNESCreate(PETSC_COMM_SELF,&nasm->subsnes[i]);CHKERRQ(ierr);
>
> these subdomains do not correspond to the "normal" one subdomain per process decomposition of the domain that you expect when you call
>
> ierr = DMDAVecGetArray(info->da,user->temp, &z);CHKERRQ(ierr); // added by rlchen
>
> inside PetscErrorCode FormFunctionLocal(DMDALocalInfo *info,Field **x,Field **f,void *ptr)
>
> In other words you are expecting that the info->da passed into FormFunctionLocal() is the same as the DA in your original main() program. But with NASM solver it simply is not, it is some other DA that is created by the NASM solver.
>
> If you want to pass some vector like your .temp vector into your local function you will need to understand exactly how the NASM solver works so you can pass the correct vectors in. Simple passing a vector associated with the original DA of the main program won't work.
>
>    Barry
>
>
>
>
>> On May 30, 2015, at 8:02 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>
>> Hi Barry and Matt,
>>
>> Many thanks for your reply. I only have one DA and I do not think I changed others when I changed the overlap.
>>
>> This error can be reproduced using the attached code (added some lines in the src/snes/examples/tutorials/ex19.c) run with:
>>
>> runex19_1:
>>         -@${MPIEXEC} -n 2 ./ex19 -da_refine 3 -snes_type nasm -da_overlap 2 -snes_monitor
>>
>> The full error messages are followed:
>> ------------------
>> lid velocity = 0.0016, prandtl # = 1, grashof # = 1
>>   0 SNES Function norm 4.066115181565e-02
>> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [1]PETSC ERROR: Arguments are incompatible
>> [1]PETSC ERROR: Vector local size 1300 is not compatible with DMDA local sizes 1400 1728
>>
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [1]PETSC ERROR: Petsc Release Version 3.5.2, unknown
>> [1]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 31 08:55:16 2015
>> [1]PETSC ERROR: Configure options --download-fblaslapack --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 --with-debugging=1 --with-shared-libraries=0
>> [1]PETSC ERROR: #1 DMDAVecGetArray() line 71 in /home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
>> [1]PETSC ERROR: #2 FormFunctionLocal() line 275 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
>> [1]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in /home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
>> [1]PETSC ERROR: #4 SNESComputeFunction() line 2033 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [1]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
>> [1]PETSC ERROR: #6 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [1]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
>> [1]PETSC ERROR: #8 SNESSolve_NASM() line 859 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
>> [1]PETSC ERROR: #9 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [1]PETSC ERROR: #10 main() line 162 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
>> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 75) - process 1
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Arguments are incompatible
>> [0]PETSC ERROR: Vector local size 1400 is not compatible with DMDA local sizes 1500 1836
>>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown
>> [0]PETSC ERROR: ./ex19 on a 64bit-debug named rlchen by rlchen Sun May 31 08:55:16 2015
>> [0]PETSC ERROR: Configure options --download-fblaslapack --download-blacs --download-scalapack --download-metis --download-parmetis --download-exodusii --download-netcdf --download-hdf5 --with-64-bit-indices --with-c2html=0 --with-mpi=1 --with-debugging=1 --with-shared-libraries=0
>> [0]PETSC ERROR: #1 DMDAVecGetArray() line 71 in /home/rlchen/soft/petsc-3.5.2/src/dm/impls/da/dagetarray.c
>> [0]PETSC ERROR: #2 FormFunctionLocal() line 275 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
>> [0]PETSC ERROR: #3 SNESComputeFunction_DMDA() line 90 in /home/rlchen/soft/petsc-3.5.2/src/snes/utils/dmdasnes.c
>> [0]PETSC ERROR: #4 SNESComputeFunction() line 2033 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 174 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [0]PETSC ERROR: #7 SNESNASMSolveLocal_Private() line 716 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
>> [0]PETSC ERROR: #8 SNESSolve_NASM() line 859 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/nasm/nasm.c
>> [0]PETSC ERROR: #9 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [0]PETSC ERROR: #10 main() line 162 in /home/rlchen/soft/petsc-3.5.2/src/snes/examples/tutorials/ex19.c
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 75) - process 0
>>
>> =====================================================================================
>> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> =   EXIT CODE: 19200
>> =   CLEANING UP REMAINING PROCESSES
>> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>> =====================================================================================
>> make: [runex19_1] Error 75 (ignored)
>> -----------------------
>>
>> Best regards,
>> Rongliang
>>
>> On 05/31/2015 01:09 AM, Barry Smith wrote:
>>>> On May 30, 2015, at 5:05 AM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>>>
>>>> Hi there,
>>>>
>>>> I am trying to use NASM to solve a nonlinear problem. In my FormFunctionLocal, I need to call DMDAVecGetArray(da, myvec, z). For the "-da_overlap 0" case, the code works fine.
>>>>
>>>> For the "-da_overlap 1" case, when I use the vector "myvec" created by DMCreateLocalVector, it works fine. But if I use the vector "myvec" created by DMCreateGlobalVector, I got the error messages:
>>>> -----------------------
>>>> [6]Arguments are incompatible
>>>> [6]PETSC ERROR: Vector local size 597870 is not compatible with DMDA local sizes 619528 664392
>>> I think you are changing something else at the same time you changed the overlap.  The number ^^^^  619528 is the local size of the global vector and it should remain the same when you change the overlap but below you have TWO other numbers 633144 and 650256  Do you have more than one DMDA?
>>> Run changing absolutely nothing except the overlap.
>>>> ----------------------
>>>>
>>>> And when I switch to "-da_overlap 2", both of the global and local version of "mrvec" got error messages:
>>>> ----------------------------------
>>>> [10]PETSC ERROR: Arguments are incompatible
>>>> [10]PETSC ERROR: Vector local size 589680 is not compatible with DMDA local sizes 633144 678680
>>>>
>>>> [3]PETSC ERROR: Arguments are incompatible
>>>> [3]PETSC ERROR: Vector local size 619528 is not compatible with DMDA local sizes 650256 696540
>>>> -----------------------------------
>>>>
>>>> Can any one tell me how to use DMDAVecGetArray in FormFunctionLocal for NASM? Thanks.
>>>>
>>>> Best regards,
>>>> Rongliang
>>>>
>> <ex19.c>


From hzhang at mcs.anl.gov  Sun May 31 09:34:08 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Sun, 31 May 2015 09:34:08 -0500
Subject: [petsc-users] MUMPS error and superLU error
In-Reply-To: <CAJ98EDoM13ezx41s+DKFik+3Xod5_v=duNJMS6BXgn+x=e_0ew@mail.gmail.com>
References: <CABzT7hNgZRH9U+rXjEYtDqtnVegLvPDVCnZVDasAbA=Lc-2UQw@mail.gmail.com>
	<CADZ9mvkihmmyMvjVDW5GV7FsFj0XPwy=o6maV+SbM4KuKZKEvQ@mail.gmail.com>
	<CABzT7hNfgcO_hknFxmVpuZx5GZUbp0mEmR6Dmmjqp7myWHOqBw@mail.gmail.com>
	<CADZ9mvkMdvfuF+=8HumJawXE9ecHAzJG1bpc0hV4-c4CS0w+1A@mail.gmail.com>
	<CABzT7hOsRq8POoopZxM=BBnxOx9yWCERCr3-=Dv6h2HVtWWaVQ@mail.gmail.com>
	<CABzT7hMWOtYDd8PtP-gULGp8Q9UN8ufnD0TFw4hvbGBjQeE3nw@mail.gmail.com>
	<CAJ98EDoM13ezx41s+DKFik+3Xod5_v=duNJMS6BXgn+x=e_0ew@mail.gmail.com>
Message-ID: <CAGCphBuBB8qt8hxSV1Ge7WbyGK3zGCHGmg6VhaXA=j_cycBN0g@mail.gmail.com>

venkatesh,

As we discussed previously, even on smaller problems,
both mumps and superlu_dist failed, although Mumps gave "OOM" error in
numerical factorization.

You acknowledged that B is singular, which may need additional
reformulation for your eigenvalue problems. The option '-st_type sinvert'
likely uses B^{-1} (have you read slepc manual?), which could be the source
of trouble.

Please investigate your model, understand why B is singular; if there is a
way to dump null space before submitting large size simulation.

Hong


On Sun, May 31, 2015 at 8:36 AM, Dave May <dave.mayhem23 at gmail.com> wrote:

> It failed due to a lack of memory. "OOM" stands for "out of memory". OOM
> killer terminated your job means you ran out of memory.
>
>
>
>
> On Sunday, 31 May 2015, venkatesh g <venkateshgk.j at gmail.com> wrote:
>
>> Hi all,
>>
>> I tried to run my Generalized Eigenproblem in 120 x 24 = 2880 cores.
>> The matrix size of A = 20GB and B = 5GB.
>>
>> It got killed after 7 Hrs of run time. Please see the mumps error log.
>> Why must it fail ?
>> I gave the command:
>>
>> aprun -n 240 -N 24 ./ex7 -f1 a110t -f2 b110t -st_type sinvert -eps_nev 1
>> -log_summary -st_ksp_type preonly -st_pc_type lu
>> -st_pc_factor_mat_solver_package mumps -mat_mumps_cntl_1 1e-2
>>
>> Kindly let me know.
>>
>> cheers,
>> Venkatesh
>>
>> On Fri, May 29, 2015 at 10:46 PM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi Matt, users,
>>>
>>> Thanks for the info. Do you also use Petsc and Slepc with MUMPS ? I get
>>> into the segmentation error if I increase my matrix size.
>>>
>>> Can you suggest other software for direct solver for QR in parallel
>>> since as LU may not be good for a singular B matrix in Ax=lambda Bx ? I am
>>> attaching the working version mumps log.
>>>
>>> My matrix size here is around 47000x47000. If I am not wrong, the memory
>>> usage per core is 272MB.
>>>
>>> Can you tell me if I am wrong ? or really if its light on memory for
>>> this matrix ?
>>>
>>> Thanks
>>> cheers,
>>> Venkatesh
>>>
>>> On Fri, May 29, 2015 at 4:00 PM, Matt Landreman <
>>> matt.landreman at gmail.com> wrote:
>>>
>>>> Dear Venkatesh,
>>>>
>>>> As you can see in the error log, you are now getting a segmentation
>>>> fault, which is almost certainly a separate issue from the info(1)=-9
>>>> memory problem you had previously. Here is one idea which may or may not
>>>> help. I've used mumps on the NERSC Edison system, and I found that I
>>>> sometimes get segmentation faults when using the default Intel compiler.
>>>> When I switched to the cray compiler the problem disappeared. So you could
>>>> perhaps try a different compiler if one is available on your system.
>>>>
>>>> Matt
>>>> On May 29, 2015 4:04 AM, "venkatesh g" <venkateshgk.j at gmail.com> wrote:
>>>>
>>>>> Hi Matt,
>>>>>
>>>>> I did what you told and read the manual of that CNTL parameters. I
>>>>> solve for that with CNTL(1)=1e-4. It is working.
>>>>>
>>>>> But it was a test matrix with size 46000x46000. Actual matrix size is
>>>>> 108900x108900 and will increase in the future.
>>>>>
>>>>> I get this error of memory allocation failed. And the binary matrix
>>>>> size of A is 20GB and B is 5 GB.
>>>>>
>>>>> Now I submit this in 240 processors each 4 GB RAM and also in 128
>>>>> Processors with total 512 GB RAM.
>>>>>
>>>>> In both the cases, it fails with the following error like memory is
>>>>> not enough. But for 90000x90000 size it had run serially in Matlab with
>>>>> <256 GB RAM.
>>>>>
>>>>> Kindly let me know.
>>>>>
>>>>> Venkatesh
>>>>>
>>>>> On Tue, May 26, 2015 at 8:02 PM, Matt Landreman <
>>>>> matt.landreman at gmail.com> wrote:
>>>>>
>>>>>> Hi Venkatesh,
>>>>>>
>>>>>> I've struggled a bit with mumps memory allocation too.  I think the
>>>>>> behavior of mumps is roughly the following. First, in the "analysis step",
>>>>>> mumps computes a minimum memory required based on the structure of nonzeros
>>>>>> in the matrix.  Then when it actually goes to factorize the matrix, if it
>>>>>> ever encounters an element smaller than CNTL(1) (default=0.01) in the
>>>>>> diagonal of a sub-matrix it is trying to factorize, it modifies the
>>>>>> ordering to avoid the small pivot, which increases the fill-in (hence
>>>>>> memory needed).  ICNTL(14) sets the margin allowed for this unanticipated
>>>>>> fill-in.  Setting ICNTL(14)=200000 as in your email is not the solution,
>>>>>> since this means mumps asks for a huge amount of memory at the start.
>>>>>> Better would be to lower CNTL(1) or (I think) use static pivoting
>>>>>> (CNTL(4)).  Read the section in the mumps manual about these CNTL
>>>>>> parameters. I typically set CNTL(1)=1e-6, which eliminated all the
>>>>>> INFO(1)=-9 errors for my problem, without having to modify ICNTL(14).
>>>>>>
>>>>>> Also, I recommend running with ICNTL(4)=3 to display diagnostics.
>>>>>> Look for the line in standard output that says "TOTAL     space in MBYTES
>>>>>> for IC factorization".  This is the amount of memory that mumps is trying
>>>>>> to allocate, and for the default ICNTL(14), it should be similar to
>>>>>> matlab's need.
>>>>>>
>>>>>> Hope this helps,
>>>>>> -Matt Landreman
>>>>>> University of Maryland
>>>>>>
>>>>>> On Tue, May 26, 2015 at 10:03 AM, venkatesh g <
>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>
>>>>>>> I posted a while ago in MUMPS forums but no one seems to reply.
>>>>>>>
>>>>>>> I am solving a large generalized Eigenvalue problem.
>>>>>>>
>>>>>>> I am getting the following error which is attached, after giving the
>>>>>>> command:
>>>>>>>
>>>>>>> /cluster/share/venkatesh/petsc-3.5.3/linux-gnu/bin/mpiexec -np 64
>>>>>>> -hosts compute-0-4,compute-0-6,compute-0-7,compute-0-8 ./ex7 -f1 a72t -f2
>>>>>>> b72t -st_type sinvert -eps_nev 3 -eps_target 0.5 -st_ksp_type preonly
>>>>>>> -st_pc_type lu -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14
>>>>>>> 200000
>>>>>>>
>>>>>>> IT IS impossible to allocate so much memory per processor.. it is
>>>>>>> asking like around 70 GB per processor.
>>>>>>>
>>>>>>> A serial job in MATLAB for the same matrices takes < 60GB.
>>>>>>>
>>>>>>> After trying out superLU_dist, I have attached the error there also
>>>>>>> (segmentation error).
>>>>>>>
>>>>>>> Kindly help me.
>>>>>>>
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150531/4519c272/attachment.html>

From bsmith at mcs.anl.gov  Sun May 31 11:58:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 31 May 2015 11:58:40 -0500
Subject: [petsc-users] Offloading linear solves in time stepper to GPU
In-Reply-To: <CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
References: <CAC0ZLSoJpepGiWiy=C4-W+0K0u5bVLOUdZDURn+xmiOpyEzQng@mail.gmail.com>
	<708950B0-5646-4237-9331-9A97339912E2@mcs.anl.gov>
	<CAC0ZLSof55_NOFNAQhHrnMNELL10eZLpb8jVQVC=5bpTRp0a3Q@mail.gmail.com>
	<871thxih75.fsf@jedbrown.org>
	<CAC0ZLSpqbcTeUVuOmpiQeV03hH88-C1C-c_XELYNpOt8Z+Ci0w@mail.gmail.com>
Message-ID: <33EEC801-94F0-45CE-AE71-7306144B1CDC@mcs.anl.gov>


> On May 30, 2015, at 10:14 PM, Harshad Sahasrabudhe <hsahasra at purdue.edu> wrote:
> 
> Is your intent to solve a problem that matters in a way that makes sense for a scientist or engineer
> 
> I want to see if we can speed up the time stepper for a large system using GPUs. For large systems with sparse matrix of size 420,000^2, each time step takes 341 sec on a single process and 180 seconds on 16 processes.

   Rather than going off on a wild goose chase it would be good to understand WHY 1) the time on one process is so poor and 2) why the speedup to 16 processes is so low. These means gathering information and then analyzing that information. 

So first you need to measure the memory bandwidth of your system for 1 to 16 processes. This is explained at http://www.mcs.anl.gov/petsc/documentation/faq.html#computers by running the "make streams NPMAX=16" then use the MPI "binding" options to see if that improves the streams numbers.   What do you get for these numbers?

Next you need to run your PETSc application with -log_summary to see how much time is in the linear solve and how what it is doing in the linear solve and how much time is spent in each part of the linear solve and how many iterations it is taking. To start run with -log_summary and 1 MPI process, 2 MPI processes, 4 MPI processes, 8 MPI processes and 16 MPI processes. What do you get for these numbers?

   In addition to 1) and 2) you need to determine what is a good preconditioner for YOUR problem. Linear iterative solvers are not black box solvers, using an inappropriate preconditioner can have many orders of magnitude difference on solution time (more than changing the hardware). If your problem is a nice elliptic operator than like -pc_type gamg might work well (or you can try the external packages -pc_type hypre or -pc_type ml ; requires installing those optional external packages; see http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html.  If your problem is a saddle-point (eg. Stokes) problem then you likely need to use the PCFIELDSPLIT preconditioner to "pull out" the saddle-point part. For more complicated simulations you will need nesting of several preconditioners.

  Barry





> So the scaling isn't that good. We also run out of memory with more number of processes. 
> 
> On Sat, May 30, 2015 at 11:01 PM, Jed Brown <jed at jedbrown.org> wrote:
> Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:
> > For now, I want to serialize the matrices and vectors and offload them to 1
> > GPU from the root process. Then distribute the result later.
> 
> Unless you have experience with these solvers and the overheads
> involved, I think you should expect this to be much slower than simply
> doing the solves using a reasonable method in the CPU.  Is your intent
> to solve a problem that matters in a way that makes sense for a
> scientist or engineer, or is it to demonstrate that a particular
> combination of packages/methods/hardware can be used?
> 


From jed at jedbrown.org  Sun May 31 19:59:44 2015
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 01 Jun 2015 02:59:44 +0200
Subject: [petsc-users] Modified Taylor-Hood elements with piece-wise
	constant pressure for Stokes equation
In-Reply-To: <CAP2=TMgqO+UZc_C8_CrA4A+8DJxXm-vDzCGEGaxuwOVFq4uWmA@mail.gmail.com>
References: <CAP2=TMhTPumTESukwLRXYOQ2-uh3cx0XuzDu5d+ve_3XOhwUkQ@mail.gmail.com>
	<877frpili3.fsf@jedbrown.org>
	<CAP2=TMgqO+UZc_C8_CrA4A+8DJxXm-vDzCGEGaxuwOVFq4uWmA@mail.gmail.com>
Message-ID: <87a8wkgs5b.fsf@jedbrown.org>

Justin Chang <jychang48 at gmail.com> writes:

> I am referring to P2 / (P1 + P0) elements, I think this is the correct way
> of expressing it. Some call it modified Taylor Hood, others call it
> something else, but it's not Crouzeix-Raviart elements.

Okay, thanks.  This pressure space is not a disjoint union (the constant
exists in both spaces) and thus the obvious "basis" is actually not
linearly independent.  I presume that people using this element do some
"pinning" (like set one cell "average" to zero) instead of enforcing a
unique expression via a Lagrange multiplier (which would involve a dense
row and column).  That may contribute to ill conditioning and in any
case, would make domain decomposition or multigrid preconditioners more
technical.  Do you know of anything explaining why the method is not
very widely used (e.g., in popular software packages, finite element
books, etc.)?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150601/da392454/attachment.pgp>